Skip to main content

Streamlining provisioning operations on VSphere with Saltstack and a vijava

In enterprise environments VMware vSphere is the most common virtualization choice.
Hence when it comes to provisioning most rely on vm templates to accelerate and standardize the creation of new virtual machines. While templates work well for both Windows and Linux vms it still leaves a lot out and administrators may have additional work that has to be carried out manually (like configure networking, optionally move the vm in the right network/resource group, create application accounts, set up IIS, tomcat, java, etc, etc).

True, most of this post-bootstrap setup can be taken care of with Puppet, Chef or, in my case, Saltstack, but what I wanted was a push-button vm provisioning that does all of the following automagically:
  1. create a new vm (only Linux vms are supported as of now)
  2. assign a pre-determined static ip address/name
  3. move vm into appropriate network
  4. configure it with Saltstack
  5. register the vm into out monitoring system (currently OpenNMS)

Overview of the process

What I have done is actually pretty simple: I have created a new Linux template and added a post boot hook in /etc/rc.local. This hook connects to a web service and downloads the appropriate configuration script for the vm.
The script is the run and does, in order, the following:
  1. resolve the virtual machine name through a dns query to find out its static ip address
  2. reconfigure networking so that eth0 uses this static ip address
  3. since that ip address is no longer valid for the vSphere network to which the vm is connected it calls another web service to move the vm's network adapter into the appropriate network
  4. the vm reboots (technically this is not necessary, as service networking restart could suffice)
  5. another post startup script (configured by the previous one) start the salt-minion daemon and waits for the operator to set state on the vm
  6. salt registers the vm with monitoring software (TBD)
Note: as the template is configured with DHCP the first time it boots up it has to connect to a restricted, DHCP-fied network used just for this initial step.

Post-boot hook

The post-boot hook is actually pretty simple:

# this goes in /etc/rc.local

/bin/mkdir /etc/salt > /dev/null 2>&1

if [ ! -e /etc/salt/firstboot ]; then
   touch /etc/salt/firstboot
   echo "Initializing server on first boot"
   ( /usr/bin/curl -s http://salt/_init | /bin/bash ) > /var/log/init 2>&1

The _init script is pretty much specific to the site, so I'll leave it out.

The vNetwork reconfiguration script

The most interesting bit in the process is perhaps the script that reconfigures the virtual machine network adapter to connect to the right network. Luckily I didn't have to use the vSphere SDK, as I could get it done with vijava instead (wrapped in a thick, delicious Groovy script).

The script itself (which is available as a gist here) is called from a web service (a python cgi, really) which additionally takes care of paramaters validation, loggin, etc.

Salt state

Salt state pretty much depends on what the vm does, but for an example see this basic state that configures snmpd on a Centos/RHEL host or check out

Register VM with OpenNMS

TBD, will cover it in a later post (hint: use provisioning).

Leave critics, suggestions or ideas in the comments.


Popular posts from this blog

From 0 to ZFS replication in 5m with syncoid

The ZFS filesystem has many features that once you try them you can never go back. One of the lesser known is probably the support for replicating a zfs filesystem by sending the changes over the network with zfs send/receive.
Technically the filesystem changes don't even need to be sent over a network: you could as well dump them on a removable disk, then receive  from the same removable disk.

Mirth: recover space when mirthdb grows out of control

I was recently asked to recover a mirth instance whose embedded database had grown to fill all available space so this is just a note-to-self kind of post.
Btw: the recovery, depending on db size and disk speed, is going to take long.

The problem A 1.8 Mirth Connect instance was started, then forgotten (well neglected, actually). The user also forgot to setup pruning so the messages filled the embedded Derby database until it grew to fill all the available space on the disk. The SO is linux.

The solution First of all: free some disk space so that the database can be started in embedded mode from the cli. You can also copy the whole mirth install to another server if you cannot free space. Depending on db size you will need a corresponding amount of space: in my case a 5GB db required around 2GB to start, process logs and then store the temp files during shrinking.

Then open a shell as the user that mirth runs as (you're not running it as root, are you?) and cd into the mirth home. …

NGINX stream module with dynamic upstreams

NGINX has had support for dynamic upstream modules for a while in the community distribution and examples abund. I think this is probably one of the clearest I could find.

Finding a similar config for stream proxies turned out to be surprisingly hard, so here I'm sharing my solution in the hope that it can be useful to somebody. Or someone more experienced can point out a better alternative.
In my case my upstream is an ELB which can and will change ip address often so using the static DNS name was not an option.