Skip to main content

Streamlining provisioning operations on VSphere with Saltstack and a vijava

In enterprise environments VMware vSphere is the most common virtualization choice.
Hence when it comes to provisioning most rely on vm templates to accelerate and standardize the creation of new virtual machines. While templates work well for both Windows and Linux vms it still leaves a lot out and administrators may have additional work that has to be carried out manually (like configure networking, optionally move the vm in the right network/resource group, create application accounts, set up IIS, tomcat, java, etc, etc).

True, most of this post-bootstrap setup can be taken care of with Puppet, Chef or, in my case, Saltstack, but what I wanted was a push-button vm provisioning that does all of the following automagically:
  1. create a new vm (only Linux vms are supported as of now)
  2. assign a pre-determined static ip address/name
  3. move vm into appropriate network
  4. configure it with Saltstack
  5. register the vm into out monitoring system (currently OpenNMS)

Overview of the process

What I have done is actually pretty simple: I have created a new Linux template and added a post boot hook in /etc/rc.local. This hook connects to a web service and downloads the appropriate configuration script for the vm.
The script is the run and does, in order, the following:
  1. resolve the virtual machine name through a dns query to find out its static ip address
  2. reconfigure networking so that eth0 uses this static ip address
  3. since that ip address is no longer valid for the vSphere network to which the vm is connected it calls another web service to move the vm's network adapter into the appropriate network
  4. the vm reboots (technically this is not necessary, as service networking restart could suffice)
  5. another post startup script (configured by the previous one) start the salt-minion daemon and waits for the operator to set state on the vm
  6. salt registers the vm with monitoring software (TBD)
Note: as the template is configured with DHCP the first time it boots up it has to connect to a restricted, DHCP-fied network used just for this initial step.

Post-boot hook

The post-boot hook is actually pretty simple:

# this goes in /etc/rc.local

/bin/mkdir /etc/salt > /dev/null 2>&1

if [ ! -e /etc/salt/firstboot ]; then
   touch /etc/salt/firstboot
   echo "Initializing server on first boot"
   ( /usr/bin/curl -s http://salt/_init | /bin/bash ) > /var/log/init 2>&1

The _init script is pretty much specific to the site, so I'll leave it out.

The vNetwork reconfiguration script

The most interesting bit in the process is perhaps the script that reconfigures the virtual machine network adapter to connect to the right network. Luckily I didn't have to use the vSphere SDK, as I could get it done with vijava instead (wrapped in a thick, delicious Groovy script).

The script itself (which is available as a gist here) is called from a web service (a python cgi, really) which additionally takes care of paramaters validation, loggin, etc.

Salt state

Salt state pretty much depends on what the vm does, but for an example see this basic state that configures snmpd on a Centos/RHEL host or check out

Register VM with OpenNMS

TBD, will cover it in a later post (hint: use provisioning).

Leave critics, suggestions or ideas in the comments.


Popular posts from this blog

Indexing Apache access logs with ELK (Elasticsearch+Logstash+Kibana)

Who said that grepping Apache logs has to be boring?

The truth is that, as Enteprise applications move to the browser too, Apache access logs are a gold mine, it does not matter what your role is: developer, support or sysadmin. If you are not mining them you are most likely missing out a ton of information and, probably, making the wrong decisions.
ELK (Elasticsearch, Logstash, Kibana) is a terrific, Open Source stack for visually analyzing Apache (or nginx) logs (but also any other timestamped data).

From 0 to ZFS replication in 5m with syncoid

The ZFS filesystem has many features that once you try them you can never go back. One of the lesser known is probably the support for replicating a zfs filesystem by sending the changes over the network with zfs send/receive.
Technically the filesystem changes don't even need to be sent over a network: you could as well dump them on a removable disk, then receive  from the same removable disk.

A not so short guide to ZFS on Linux

Updated Oct 16 2013: shadow copies, memory settings and links for further learning.
Updated Nov 15 2013: shadow copies example, samba tuning.

Unless you've been living under a rock you should have by now heard many stories about how awesome ZFS is and the many ways it can help with saving your bacon.

The downside is that ZFS is not available (natively) for Linux because the CDDL license under which it is released is incompatible with the GPL. Assuming you are not interested in converting to one of the many Illumos distributions or FreeBSD this guide might serve you as a starting point if you are attracted  by ZFS features but are reluctant to try it out on production systems.

Basically in this post I note down both the tought process and the actual commands for implementing a fileserver for a small office. The fileserver will run as a virtual machine in a large ESXi host and use ZFS as the filesystem for shared data.