Streamlining provisioning operations on VSphere with Saltstack and a vijava

In enterprise environments VMware vSphere is the most common virtualization choice.
Hence when it comes to provisioning most rely on vm templates to accelerate and standardize the creation of new virtual machines. While templates work well for both Windows and Linux vms it still leaves a lot out and administrators may have additional work that has to be carried out manually (like configure networking, optionally move the vm in the right network/resource group, create application accounts, set up IIS, tomcat, java, etc, etc).

True, most of this post-bootstrap setup can be taken care of with Puppet, Chef or, in my case, Saltstack, but what I wanted was a push-button vm provisioning that does all of the following automagically:

1. create a new vm (only Linux vms are supported as of now)
2. assign a pre-determined static ip address/name
3. move vm into appropriate network
4. configure it with Saltstack
5. register the vm into out monitoring system (currently OpenNMS)

Overview of the process

What I have done is actually pretty simple: I have created a new Linux template and added a post boot hook in /etc/rc.local. This hook connects to a web service and downloads the appropriate configuration script for the vm.

The script is the run and does, in order, the following:

1. resolve the virtual machine name through a dns query to find out its static ip address
2. reconfigure networking so that eth0 uses this static ip address
3. since that ip address is no longer valid for the vSphere network to which the vm is connected it calls another web service to move the vm's network adapter into the appropriate network
4. the vm reboots (technically this is not necessary, as service networking restart could suffice)
5. another post startup script (configured by the previous one) start the salt-minion daemon and waits for the operator to set state on the vm
6. salt registers the vm with monitoring software (TBD)

Note: as the template is configured with DHCP the first time it boots up it has to connect to a restricted, DHCP-fied network used just for this initial step.

Post-boot hook

The post-boot hook is actually pretty simple:

#
# this goes in /etc/rc.local
#

/bin/mkdir /etc/salt > /dev/null 2>&1

if [ ! -e /etc/salt/firstboot ]; then
   touch /etc/salt/firstboot
   echo "Initializing server on first boot"
   ( /usr/bin/curl -s http://salt/_init | /bin/bash ) > /var/log/init 2>&1
fi 

The _init script is pretty much specific to the site, so I'll leave it out.

The vNetwork reconfiguration script

The most interesting bit in the process is perhaps the script that reconfigures the virtual machine network adapter to connect to the right network. Luckily I didn't have to use the vSphere SDK, as I could get it done with vijava instead (wrapped in a thick, delicious Groovy script).

The script itself (which is available as a gist here) is called from a web service (a python cgi, really) which additionally takes care of paramaters validation, loggin, etc.

Salt state

Salt state pretty much depends on what the vm does, but for an example see this basic state that configures snmpd on a Centos/RHEL host or check out http://saltstarters.org/.

Register VM with OpenNMS

TBD, will cover it in a later post (hint: use provisioning).

Leave critics, suggestions or ideas in the comments.