Skip to main content

Salt diaries: states (part 2 of deploying salt on a small network)

After part 1 of this series I had Salt running properly on all minions. It's now time to get some work done with it.

We will start with something simple like making sure that ntp is installed and running on all minions. In order to to do that we will use the Salt states enforcement feature.
The default salt states configuration requires that:

  1. state definitions be kept in /srv/salt
  2. the default state be named top.sls

We will probably need to create both the directory and the files, which we can do with the following command (check that you are not overwriting your own state, needs to be done on the master only!):

mkdir -p /srv/salt
cat <<EOF >/srv/salt/top.sls
    - ntp

What this state definition means is that the base state requires all nodes (as selected by '*') to apply the ntp state. Since we have not yet defined an ntp state we are going to do it right away:

cat <<EOF >/srv/salt/ntp.sls
    - installed
    - name: ntpd
    - running
    - require:
      - pkg: ntp

The ntp state basically instructs salt to check that all nodes have a package called ntp installed and a service called ntpd running. Note that the service check requires the package to be installed: this is to make sure that salt does not try to start the service before the package has been installed, which, of course, is guaranteed to fail.
Also note that we had to specify the name of the service because in CentOS the service is named differently from the package. If we didn't salt would have looked for a service called ntp as the state name implies.
Before we apply the states we can ask salt to report the to-be-applied state for all nodes. (Note: in my case I had ntp installed on some nodes, but not all of them) The command to use is:

salt '*' state.show_highstate

Link to the state commands reference.

If all is well, we can now apply the state to all nodes (to apply it to just one node use the fqdn node name instead of *):

salt -t 60 '*' state.highstate

Tip: it is probably a good idea to raise the timeout when possibly long operations like these need to be performed. When I first tried without the longer timeout some nodes became unresponsive to the master and had to the restart the salt-master process.

As the states are applied the command will output the changes for each node. Running the same command again after all nodes state has been sync'ed should not report any change.

Considerations on states definitions

In this example I could have embedded the ntp state definition in the top.sls file. I preferred not to because
  1. the top.sls file will quickly grow out of control as states are embedded into it
  2. the top.sls file imho should only be used as an entry point where node selectors and states are matched to each other. To know which states go to which nodes one only has to look into this file. On the other hand, scattering selectors and definitions among many files will quickly make the whole system unmanageable.
    Note: grain selectors still can and must be used inside individual substate definitions to account for different OS, features, etc.

Use a version control system for salt states definitions

One final (important) suggestion I have is to use git or svn or any other version control system you might be familiar with to keep track of state changes, especially if you are managing the Salt server together with colleagues. In that case you will also want to make sure that each person uses their own account and not just 'root'.

Next steps

In the next posts I am going to deploy salt on SLES servers and modify the ntp state to account for the difference in configuration and to add many more states (one that I am thinking of is a configuration for a centralized syslog server).

See all my Salt-related posts


Popular posts from this blog

Indexing Apache access logs with ELK (Elasticsearch+Logstash+Kibana)

Who said that grepping Apache logs has to be boring?

The truth is that, as Enteprise applications move to the browser too, Apache access logs are a gold mine, it does not matter what your role is: developer, support or sysadmin. If you are not mining them you are most likely missing out a ton of information and, probably, making the wrong decisions.
ELK (Elasticsearch, Logstash, Kibana) is a terrific, Open Source stack for visually analyzing Apache (or nginx) logs (but also any other timestamped data).

From 0 to ZFS replication in 5m with syncoid

The ZFS filesystem has many features that once you try them you can never go back. One of the lesser known is probably the support for replicating a zfs filesystem by sending the changes over the network with zfs send/receive.
Technically the filesystem changes don't even need to be sent over a network: you could as well dump them on a removable disk, then receive  from the same removable disk.

Mirth: recover space when mirthdb grows out of control

I was recently asked to recover a mirth instance whose embedded database had grown to fill all available space so this is just a note-to-self kind of post.
Btw: the recovery, depending on db size and disk speed, is going to take long.

The problem A 1.8 Mirth Connect instance was started, then forgotten (well neglected, actually). The user also forgot to setup pruning so the messages filled the embedded Derby database until it grew to fill all the available space on the disk. The SO is linux.

The solution First of all: free some disk space so that the database can be started in embedded mode from the cli. You can also copy the whole mirth install to another server if you cannot free space. Depending on db size you will need a corresponding amount of space: in my case a 5GB db required around 2GB to start, process logs and then store the temp files during shrinking.

Then open a shell as the user that mirth runs as (you're not running it as root, are you?) and cd into the mirth home. …