Skip to main content

A not so short guide to TDD SaltStack formulas

One of the hardest parts about Infrastructure As Code and Configuration Management is establishing a discipline on developing, testing and deploying changes.
Developers follow established practices and tools have been built and perfected over the last decade and a half. On the other hand sysadmins and ops people do not have the same tooling and culture because estensive automation has only become a trend recently.

So if Infrastructure As Code allows you to version the infrastructure your code runs on, what good is it if then there are no tools or established practices to follow?

Luckily the situation is changing and in this post I'm outlining a methodology for test driven development of SaltStack Formulas.

The idea is that with a single command you can run your formula against a matrix of platforms (operating systems) and suites (or configurations). Each cell of the matrix will be tested and the result is a build failure or success much alike to what every half-decent developer of this world sees every day. The same matrix can also be spun up on a variety of providers (EC2, DigitalOcean, OpenStack, etc), which is useful for CI servers.

We will use Vagrant/Virtualbox for provisioning the various combinations of platforms and suites. We will also pull a couple of tricks to speed up the builds by caching as much as possible on the local machine.

In general I would avoid using containers instead of a full VM, because of their limitations in emulating a full OS. YMMV, but chances are you will hit problems very early in your tests.

One thing to remember is that in this (long-ish) post I will be mentioning half a dozen tools.
Fortunately for us someone already did all the hard work in pulling them together into one tool so you don't need to do what each tool does. It does help though to familiarize with some of them and the toolchain in general. I provide a list at the bottom.
That umbrella tool that drives all is called Test Kitchen. Test Kitchen was originally developed for Chef, but it eventually got support for Salt with a plugin.

Before we start you should have the following ready and working. If not stop right here and come back after you got to know your way around them:
  1. a Linux or OSX host
  2. Vagrant and Virtualbox installed and functioning
  3. Ruby >= 1.9.3p484
  4. Python >= 2.7
Any Ubuntu >= 12.04 will do. I have Ubuntu 12.04 LTS, just in case you wonder.


First thing we install is this python module: saltscaffold. Instructions are on the README (at the time of this writing). Once it's installed we will use it to generate all the boilerplate for our spanking new formula. I'll call it myservice:

saltscaffold -p myservice

If there are no errors cd into myservice-formula and take a moment to inspect the files. In myservice you will find the actual files that define our formula, while in the folder named test we find the integration tests. We will come back to them later. Look at .kitchen.yml which will drive our local tests and .kitchen-ci.yml which can be used in a CI environment. Inspect platforms and suites.

At this point we need to run a first batch of integration tests locally to make sure we are starting from something sane. To run them we should type kitchen verify, but thats is likely not to work, because we haven't installed it yet. So let's fix that. As root or sudo:

gem install test-kitchen
gem install kitchen-vagrant
gem install kitchen-salt

If that goes well then run your first integration test. But before that you should know that you are about to download the whole internet (well not really, just ~ 500MB of it) into your computer. It will be better after the first time but it will still take time. I will show how to roughly cut the time in half later in this post. If you have a slow internet connection (<10Mb) then find something else (coffee?) to do in the meanwhile:

kitchen verify

What this does is:
  1. build 2 VMs running Debian 8.2 (2 is the number of suites by the number of platforms, remember?) on your computer
  2. provision salt into the with salt-bootstrap
  3. apply the formula
  4. run serverspec against each VM
  5. report
If you run kitchen verify again it will just run steps 3,4,5 so the execution will actually be very fast, almost immediate on my computer.

Now that we have all the basic scaffolding in place we can develop our formula and test it with kitchen verify. Red, green, refactor. Git add, commit and push. You know the drill.

Here are a few other useful commands. To test the formula in a fresh VM run:

kitchen destroy
kitchen verify

To log into a vm type:

kitchen login <vm name>

To get the name of the VMs:

kitchen list

Speeding up things

Generally speaking people love Vagrant, when they first discover it. The love then fades when they need to run something iteratively (like tests) which often implies long times waiting for apt or yum.

But a proven fact in IT is that whenever there is an itch there is someone scratching and the scratching more often than not leads to a cure. In our case there is a Vagrant plugin called cachier (repo) which can be used to cache yum/apt data, ruby gems, npm, you name it.
Install it with:

vagrant plugin install vagrant-cachier

Then configure vagrant to use cachier for all your boxes by editing ~/.vagrant.d/Vagrantfile (create if it does not exists):

Vagrant.configure("2") do |config|
  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box
    config.cache.enable :generic, { :cache_dir => "/var/cache/generic" }
    config.cache.enable :gem

Then edit .kitchen.yml and activate caching for the chef package by setting the omnibus_cachier flag on the provisioner block:

Destroy and recreate the VMs to enable cachier on the newly built images.

Links and references

  1. Kitchen CI:
  2. Inspiration and good insight on a SaltStack workflow:
  3. The Test Kitchen SaltStack provisioner:
  4. Ideas on how to get rid of the Chef download (speed up tests):
  5. The complete list of Test Kitchen gems:
  6. Serverspec reference:
  7. Infratest (a python alternative to serverspec):


Popular posts from this blog

From 0 to ZFS replication in 5m with syncoid

The ZFS filesystem has many features that once you try them you can never go back. One of the lesser known is probably the support for replicating a zfs filesystem by sending the changes over the network with zfs send/receive.
Technically the filesystem changes don't even need to be sent over a network: you could as well dump them on a removable disk, then receive  from the same removable disk.

Mirth: recover space when mirthdb grows out of control

I was recently asked to recover a mirth instance whose embedded database had grown to fill all available space so this is just a note-to-self kind of post.
Btw: the recovery, depending on db size and disk speed, is going to take long.

The problem A 1.8 Mirth Connect instance was started, then forgotten (well neglected, actually). The user also forgot to setup pruning so the messages filled the embedded Derby database until it grew to fill all the available space on the disk. The SO is linux.

The solution First of all: free some disk space so that the database can be started in embedded mode from the cli. You can also copy the whole mirth install to another server if you cannot free space. Depending on db size you will need a corresponding amount of space: in my case a 5GB db required around 2GB to start, process logs and then store the temp files during shrinking.

Then open a shell as the user that mirth runs as (you're not running it as root, are you?) and cd into the mirth home. …

Indexing Apache access logs with ELK (Elasticsearch+Logstash+Kibana)

Who said that grepping Apache logs has to be boring?

The truth is that, as Enteprise applications move to the browser too, Apache access logs are a gold mine, it does not matter what your role is: developer, support or sysadmin. If you are not mining them you are most likely missing out a ton of information and, probably, making the wrong decisions.
ELK (Elasticsearch, Logstash, Kibana) is a terrific, Open Source stack for visually analyzing Apache (or nginx) logs (but also any other timestamped data).