Skip to main content

A not so short guide to TDD SaltStack formulas

One of the hardest parts about Infrastructure As Code and Configuration Management is establishing a discipline on developing, testing and deploying changes.
Developers follow established practices and tools have been built and perfected over the last decade and a half. On the other hand sysadmins and ops people do not have the same tooling and culture because estensive automation has only become a trend recently.

So if Infrastructure As Code allows you to version the infrastructure your code runs on, what good is it if then there are no tools or established practices to follow?

Luckily the situation is changing and in this post I'm outlining a methodology for test driven development of SaltStack Formulas.

The idea is that with a single command you can run your formula against a matrix of platforms (operating systems) and suites (or configurations). Each cell of the matrix will be tested and the result is a build failure or success much alike to what every half-decent developer of this world sees every day. The same matrix can also be spun up on a variety of providers (EC2, DigitalOcean, OpenStack, etc), which is useful for CI servers.

We will use Vagrant/Virtualbox for provisioning the various combinations of platforms and suites. We will also pull a couple of tricks to speed up the builds by caching as much as possible on the local machine.

In general I would avoid using containers instead of a full VM, because of their limitations in emulating a full OS. YMMV, but chances are you will hit problems very early in your tests.

One thing to remember is that in this (long-ish) post I will be mentioning half a dozen tools.
Fortunately for us someone already did all the hard work in pulling them together into one tool so you don't need to do what each tool does. It does help though to familiarize with some of them and the toolchain in general. I provide a list at the bottom.
That umbrella tool that drives all is called Test Kitchen. Test Kitchen was originally developed for Chef, but it eventually got support for Salt with a plugin.

Before we start you should have the following ready and working. If not stop right here and come back after you got to know your way around them:
  1. a Linux or OSX host
  2. Vagrant and Virtualbox installed and functioning
  3. Ruby >= 1.9.3p484
  4. Python >= 2.7
Any Ubuntu >= 12.04 will do. I have Ubuntu 12.04 LTS, just in case you wonder.


First thing we install is this python module: saltscaffold. Instructions are on the README (at the time of this writing). Once it's installed we will use it to generate all the boilerplate for our spanking new formula. I'll call it myservice:

saltscaffold -p myservice

If there are no errors cd into myservice-formula and take a moment to inspect the files. In myservice you will find the actual files that define our formula, while in the folder named test we find the integration tests. We will come back to them later. Look at .kitchen.yml which will drive our local tests and .kitchen-ci.yml which can be used in a CI environment. Inspect platforms and suites.

At this point we need to run a first batch of integration tests locally to make sure we are starting from something sane. To run them we should type kitchen verify, but thats is likely not to work, because we haven't installed it yet. So let's fix that. As root or sudo:

gem install test-kitchen
gem install kitchen-vagrant
gem install kitchen-salt

If that goes well then run your first integration test. But before that you should know that you are about to download the whole internet (well not really, just ~ 500MB of it) into your computer. It will be better after the first time but it will still take time. I will show how to roughly cut the time in half later in this post. If you have a slow internet connection (<10Mb) then find something else (coffee?) to do in the meanwhile:

kitchen verify

What this does is:
  1. build 2 VMs running Debian 8.2 (2 is the number of suites by the number of platforms, remember?) on your computer
  2. provision salt into the with salt-bootstrap
  3. apply the formula
  4. run serverspec against each VM
  5. report
If you run kitchen verify again it will just run steps 3,4,5 so the execution will actually be very fast, almost immediate on my computer.

Now that we have all the basic scaffolding in place we can develop our formula and test it with kitchen verify. Red, green, refactor. Git add, commit and push. You know the drill.

Here are a few other useful commands. To test the formula in a fresh VM run:

kitchen destroy
kitchen verify

To log into a vm type:

kitchen login <vm name>

To get the name of the VMs:

kitchen list

Speeding up things

Generally speaking people love Vagrant, when they first discover it. The love then fades when they need to run something iteratively (like tests) which often implies long times waiting for apt or yum.

But a proven fact in IT is that whenever there is an itch there is someone scratching and the scratching more often than not leads to a cure. In our case there is a Vagrant plugin called cachier (repo) which can be used to cache yum/apt data, ruby gems, npm, you name it.
Install it with:

vagrant plugin install vagrant-cachier

Then configure vagrant to use cachier for all your boxes by editing ~/.vagrant.d/Vagrantfile (create if it does not exists):

Vagrant.configure("2") do |config|
  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :box
    config.cache.enable :generic, { :cache_dir => "/var/cache/generic" }
    config.cache.enable :gem

Then edit .kitchen.yml and activate caching for the chef package by setting the omnibus_cachier flag on the provisioner block:

Destroy and recreate the VMs to enable cachier on the newly built images.

Links and references

  1. Kitchen CI:
  2. Inspiration and good insight on a SaltStack workflow:
  3. The Test Kitchen SaltStack provisioner:
  4. Ideas on how to get rid of the Chef download (speed up tests):
  5. The complete list of Test Kitchen gems:
  6. Serverspec reference:
  7. Infratest (a python alternative to serverspec):


Popular posts from this blog

Indexing Apache access logs with ELK (Elasticsearch+Logstash+Kibana)

Who said that grepping Apache logs has to be boring?

The truth is that, as Enteprise applications move to the browser too, Apache access logs are a gold mine, it does not matter what your role is: developer, support or sysadmin. If you are not mining them you are most likely missing out a ton of information and, probably, making the wrong decisions.
ELK (Elasticsearch, Logstash, Kibana) is a terrific, Open Source stack for visually analyzing Apache (or nginx) logs (but also any other timestamped data).

From 0 to ZFS replication in 5m with syncoid

The ZFS filesystem has many features that once you try them you can never go back. One of the lesser known is probably the support for replicating a zfs filesystem by sending the changes over the network with zfs send/receive.
Technically the filesystem changes don't even need to be sent over a network: you could as well dump them on a removable disk, then receive  from the same removable disk.

A not so short guide to ZFS on Linux

Updated Oct 16 2013: shadow copies, memory settings and links for further learning.
Updated Nov 15 2013: shadow copies example, samba tuning.

Unless you've been living under a rock you should have by now heard many stories about how awesome ZFS is and the many ways it can help with saving your bacon.

The downside is that ZFS is not available (natively) for Linux because the CDDL license under which it is released is incompatible with the GPL. Assuming you are not interested in converting to one of the many Illumos distributions or FreeBSD this guide might serve you as a starting point if you are attracted  by ZFS features but are reluctant to try it out on production systems.

Basically in this post I note down both the tought process and the actual commands for implementing a fileserver for a small office. The fileserver will run as a virtual machine in a large ESXi host and use ZFS as the filesystem for shared data.