A not so short guide to TDD SaltStack formulas
One of the hardest parts about Infrastructure As Code and Configuration Management is establishing a discipline on developing, testing and deploying changes.
Developers follow established practices and tools have been built and perfected over the last decade and a half. On the other hand sysadmins and ops people do not have the same tooling and culture because estensive automation has only become a trend recently.
So if Infrastructure As Code allows you to version the infrastructure your code runs on, what good is it if then there are no tools or established practices to follow?
Luckily the situation is changing and in this post I'm outlining a methodology for test driven development of SaltStack Formulas.
The idea is that with a single command you can run your formula against a matrix of platforms (operating systems) and suites (or configurations). Each cell of the matrix will be tested and the result is a build failure or success much alike to what every half-decent developer of this world sees every day. The same matrix can also be spun up on a variety of providers (EC2, DigitalOcean, OpenStack, etc), which is useful for CI servers.
We will use Vagrant/Virtualbox for provisioning the various combinations of platforms and suites. We will also pull a couple of tricks to speed up the builds by caching as much as possible on the local machine.
In general I would avoid using containers instead of a full VM, because of their limitations in emulating a full OS. YMMV, but chances are you will hit problems very early in your tests.
One thing to remember is that in this (long-ish) post I will be mentioning half a dozen tools.
Fortunately for us someone already did all the hard work in pulling them together into one tool so you don't need to do what each tool does. It does help though to familiarize with some of them and the toolchain in general. I provide a list at the bottom.
That umbrella tool that drives all is called Test Kitchen. Test Kitchen was originally developed for Chef, but it eventually got support for Salt with a plugin.
Before we start you should have the following ready and working. If not stop right here and come back after you got to know your way around them:
Then edit .kitchen.yml and activate caching for the chef package by setting the omnibus_cachier flag on the provisioner block:
Destroy and recreate the VMs to enable cachier on the newly built images.
Developers follow established practices and tools have been built and perfected over the last decade and a half. On the other hand sysadmins and ops people do not have the same tooling and culture because estensive automation has only become a trend recently.
So if Infrastructure As Code allows you to version the infrastructure your code runs on, what good is it if then there are no tools or established practices to follow?
Luckily the situation is changing and in this post I'm outlining a methodology for test driven development of SaltStack Formulas.
The idea is that with a single command you can run your formula against a matrix of platforms (operating systems) and suites (or configurations). Each cell of the matrix will be tested and the result is a build failure or success much alike to what every half-decent developer of this world sees every day. The same matrix can also be spun up on a variety of providers (EC2, DigitalOcean, OpenStack, etc), which is useful for CI servers.
We will use Vagrant/Virtualbox for provisioning the various combinations of platforms and suites. We will also pull a couple of tricks to speed up the builds by caching as much as possible on the local machine.
In general I would avoid using containers instead of a full VM, because of their limitations in emulating a full OS. YMMV, but chances are you will hit problems very early in your tests.
One thing to remember is that in this (long-ish) post I will be mentioning half a dozen tools.
Fortunately for us someone already did all the hard work in pulling them together into one tool so you don't need to do what each tool does. It does help though to familiarize with some of them and the toolchain in general. I provide a list at the bottom.
That umbrella tool that drives all is called Test Kitchen. Test Kitchen was originally developed for Chef, but it eventually got support for Salt with a plugin.
Before we start you should have the following ready and working. If not stop right here and come back after you got to know your way around them:
- a Linux or OSX host
- Vagrant and Virtualbox installed and functioning
- Ruby >= 1.9.3p484
- Python >= 2.7
Any Ubuntu >= 12.04 will do. I have Ubuntu 12.04 LTS, just in case you wonder.
Installation
First thing we install is this python module: saltscaffold. Instructions are on the README (at the time of this writing). Once it's installed we will use it to generate all the boilerplate for our spanking new formula. I'll call it myservice:
saltscaffold -p myservice
If there are no errors cd into myservice-formula and take a moment to inspect the files. In myservice you will find the actual files that define our formula, while in the folder named test we find the integration tests. We will come back to them later. Look at .kitchen.yml which will drive our local tests and .kitchen-ci.yml which can be used in a CI environment. Inspect platforms and suites.
At this point we need to run a first batch of integration tests locally to make sure we are starting from something sane. To run them we should type kitchen verify, but thats is likely not to work, because we haven't installed it yet. So let's fix that. As root or sudo:
gem install test-kitchen gem install kitchen-vagrant gem install kitchen-salt
If that goes well then run your first integration test. But before that you should know that you are about to download the whole internet (well not really, just ~ 500MB of it) into your computer. It will be better after the first time but it will still take time. I will show how to roughly cut the time in half later in this post. If you have a slow internet connection (<10Mb) then find something else (coffee?) to do in the meanwhile:
kitchen verify
What this does is:
- build 2 VMs running Debian 8.2 (2 is the number of suites by the number of platforms, remember?) on your computer
- provision salt into the with salt-bootstrap
- apply the formula
- run serverspec against each VM
- report
If you run kitchen verify again it will just run steps 3,4,5 so the execution will actually be very fast, almost immediate on my computer.
Now that we have all the basic scaffolding in place we can develop our formula and test it with kitchen verify. Red, green, refactor. Git add, commit and push. You know the drill.
Here are a few other useful commands. To test the formula in a fresh VM run:
To log into a vm type:
To get the name of the VMs:
But a proven fact in IT is that whenever there is an itch there is someone scratching and the scratching more often than not leads to a cure. In our case there is a Vagrant plugin called cachier (repo) which can be used to cache yum/apt data, ruby gems, npm, you name it.
Install it with:
Then configure vagrant to use cachier for all your boxes by editing ~/.vagrant.d/Vagrantfile (create if it does not exists):
Here are a few other useful commands. To test the formula in a fresh VM run:
kitchen destroy kitchen verify
To log into a vm type:
kitchen login <vm name>
To get the name of the VMs:
kitchen list
Speeding up things
Generally speaking people love Vagrant, when they first discover it. The love then fades when they need to run something iteratively (like tests) which often implies long times waiting for apt or yum.But a proven fact in IT is that whenever there is an itch there is someone scratching and the scratching more often than not leads to a cure. In our case there is a Vagrant plugin called cachier (repo) which can be used to cache yum/apt data, ruby gems, npm, you name it.
Install it with:
vagrant plugin install vagrant-cachier
Then configure vagrant to use cachier for all your boxes by editing ~/.vagrant.d/Vagrantfile (create if it does not exists):
Vagrant.configure("2") do |config|
if Vagrant.has_plugin?("vagrant-cachier")
config.cache.scope = :box
config.cache.enable :generic, { :cache_dir => "/var/cache/generic" }
config.cache.enable :gem
end
end
Then edit .kitchen.yml and activate caching for the chef package by setting the omnibus_cachier flag on the provisioner block:
Destroy and recreate the VMs to enable cachier on the newly built images.
Links and references
- Kitchen CI: http://kitchen.ci/
- Inspiration and good insight on a SaltStack workflow: https://www.reddit.com/r/saltstack/comments/4ebic5/what_does_your_workflow_with_saltstack_look_like/
- The Test Kitchen SaltStack provisioner: https://github.com/simonmcc/kitchen-salt
- Ideas on how to get rid of the Chef download (speed up tests): https://github.com/simonmcc/kitchen-salt/issues/35
- The complete list of Test Kitchen gems: https://rubygems.org/search?utf8=%E2%9C%93&query=kitchen
- Serverspec reference: http://serverspec.org/resource_types.html
- Infratest (a python alternative to serverspec): https://github.com/philpep/testinfra