Friday, August 24, 2012

Salt Diaries: installing on SLES (episode 3)

Welcome to the third episode of the series! In the previous posts we have installed salt on CentOS machines and then moved on with a basic state configuration (we will cover more in the coming postst).

Now it's time to handle those pesky SLES hosts for which there are no pre built binaries. Therefore we'll have to install salt using pip.
I'll cover SLES11 in this post as that's the only variant I have. Hopefully other versions should require only minor changes.

Note: active subscription to Novell update service is required as the following packages can only be found on SLES 11 SDK (it's an iso, and a large one, so if you don't have it around start downloading it before you start): python-devel libopenssl-devel zlib-devel swig

Installation

Add the SDK iso in the Software Management sources. Then, as root, run the following commands (answer yes when required):

zypper in gcc-c++ python-devel libopenssl-devel zlib-devel swig
zypper -p http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/SLE_11_SP1/ -v in zeromq
curl http://python-distribute.org/distribute_setup.py | python
curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python
pip install -U salt

Running salt-minion

Salt programs (salt-minion, salt, salt-key, etc) should now be installed. Since pip does not provide init scripts or configuration files, we'll have to handle that ourselves. For the configuration file there are lots of options: it can be copied from other minions, can be totally dispensed with (if your salt master is resolvable as salt) o can be as minimalistic as:

log_file: /var/log/salt/minion
master: your-master-hostname


In my case I went for the minimalistic option and created the minion file with just the lines above in /etc/salt.
To have salt-minion start and stop as a service I copied the /etc/init.d/skeleton template in /etc/init.d/salt-minion and customized it.
The edited version is available in this gist (remember to chmod it if you copy it from the gist).

The service can then be activated with:
chkconfig salt-minion on

Reboot and enjoy.

Thursday, August 23, 2012

Mirth: recover space when mirthdb grows out of control

I was recently asked to recover a mirth instance whose embedded database had grown to fill all available space so this is just a note-to-self kind of post.
Btw: the recovery, depending on db size and disk speed, is going to take long.

The problem

A 1.8 Mirth Connect instance was started, then forgotten (well neglected, actually). The user also forgot to setup pruning so the messages filled the embedded Derby database until it grew to fill all the available space on the disk.
The SO is linux.

The solution

First of all: free some disk space so that the database can be started in embedded mode from the cli. You can also copy the whole mirth install to another server if you cannot free space. Depending on db size you will need a corresponding amount of space: in my case a 5GB db required around 2GB to start, process logs and then store the temp files during shrinking.

Then open a shell as the user that mirth runs as (you're not running it as root, are you?) and cd into the mirth home.
Start the Derby cli as follows:

java -cp lib/derbytools.jar:lib/derby.jar org.apache.derby.tools.ij

All the following commands were given on the Derby console:

ij> connect 'jdbc:derby:mirthdb';
ij> set schema app;
ij> delete from message where DATE_CREATED < TIMESTAMP('2012-01-01 00:00:00');
ij> call SYSCS_UTIL.SYSCS_COMPRESS_TABLE('APP', 'MESSAGE', 1);
ij> exit;

(Basically these commands delete messages older than this year and then compact the stable) At this point the database whould have shrunk enough to restart Mirth and enable the pruner to keep at the desired size.

Last, a word to the wise: embedded db is not recommended for production use.

Sunday, August 19, 2012

Salt diaries: states (part 2 of deploying salt on a small network)

After part 1 of this series I had Salt running properly on all minions. It's now time to get some work done with it.

We will start with something simple like making sure that ntp is installed and running on all minions. In order to to do that we will use the Salt states enforcement feature.
The default salt states configuration requires that:

  1. state definitions be kept in /srv/salt
  2. the default state be named top.sls

We will probably need to create both the directory and the files, which we can do with the following command (check that you are not overwriting your own state, needs to be done on the master only!):

mkdir -p /srv/salt
cat <<EOF >/srv/salt/top.sls
base:
  '*':
    - ntp
EOF

What this state definition means is that the base state requires all nodes (as selected by '*') to apply the ntp state. Since we have not yet defined an ntp state we are going to do it right away:

cat <<EOF >/srv/salt/ntp.sls
ntp:
  pkg:
    - installed
  service:
    - name: ntpd
    - running
    - require:
      - pkg: ntp
EOF

The ntp state basically instructs salt to check that all nodes have a package called ntp installed and a service called ntpd running. Note that the service check requires the package to be installed: this is to make sure that salt does not try to start the service before the package has been installed, which, of course, is guaranteed to fail.
Also note that we had to specify the name of the service because in CentOS the service is named differently from the package. If we didn't salt would have looked for a service called ntp as the state name implies.
Before we apply the states we can ask salt to report the to-be-applied state for all nodes. (Note: in my case I had ntp installed on some nodes, but not all of them) The command to use is:

salt '*' state.show_highstate

Link to the state commands reference.

If all is well, we can now apply the state to all nodes (to apply it to just one node use the fqdn node name instead of *):

salt -t 60 '*' state.highstate

Tip: it is probably a good idea to raise the timeout when possibly long operations like these need to be performed. When I first tried without the longer timeout some nodes became unresponsive to the master and had to the restart the salt-master process.

As the states are applied the command will output the changes for each node. Running the same command again after all nodes state has been sync'ed should not report any change.

Considerations on states definitions

In this example I could have embedded the ntp state definition in the top.sls file. I preferred not to because
  1. the top.sls file will quickly grow out of control as states are embedded into it
  2. the top.sls file imho should only be used as an entry point where node selectors and states are matched to each other. To know which states go to which nodes one only has to look into this file. On the other hand, scattering selectors and definitions among many files will quickly make the whole system unmanageable.
    Note: grain selectors still can and must be used inside individual substate definitions to account for different OS, features, etc.

Use a version control system for salt states definitions

One final (important) suggestion I have is to use git or svn or any other version control system you might be familiar with to keep track of state changes, especially if you are managing the Salt server together with colleagues. In that case you will also want to make sure that each person uses their own account and not just 'root'.

Next steps

In the next posts I am going to deploy salt on SLES servers and modify the ntp state to account for the difference in configuration and to add many more states (one that I am thinking of is a configuration for a centralized syslog server).

See all my Salt-related posts

Wednesday, August 08, 2012

Salt diaries: deploying salt on a small network

This post is the first in a series documenting the deployment of Salt on a small network ( ~ 100 hosts, initially targeting only linux-based ones which account for roughly half of it).

Due to the low number of hosts I have gone for a single master layout. The linux hosts are for the greatest part running Centos 5.[4,5] in both x86 and x64 favors, and just a couple running SLES.

Installing salt master

The easiest way to install salt on Centos is to pull in the epel repository :

rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

then install salt with yum:

yum install -y salt-master

Since minions by default will attempt to connect to the salt master by resolving an host named salt I configured a salt cname record for the salt master host in the dns server. At this point the master can be started with:

/etc/init.d/salt-master start

Note: I don't have firewall or SELinux enabled. In particular SELinux is problaly not yet supported at all.

Installing salt minions

The procedure for minions is basically the same for master with the difference that the package to install in salt-minion instead of salt-master:

yum install -y salt-minion && /etc/init.d/salt-minion start

Moving back to the master, the salt-key command can be used to check that minions have connected to the it and their keys are pending for acceptance. In a couple of cases the minions reported localhost.localdomain instead of the correct hostname. To fix it I had to edit /etc/hosts on the minion, remove the real hostname (in both unqualified and qualified form) from 127.0.0.1 and ::1 lines, and then restart salt-minion.
The mismatched key can be removed from the master with:

salt-key -r localhost.localdomain

Testing

Before moving on I wanted to make sure that everything is working as expected, so I ran this command on the master:

salt -v '*' test.ping

If the minions are running correctly you should get a True response as each minion attempts to ping the master. I used the -v option so that the master reports minions that did not respond. If some of your minions are busy and/or on slow networks consider raising the timeout with:

salt -t 60 -v '*' test.ping

That's it for now, in the next post I will get a basic states configuration working to make sure that all minions have a minimum configuration applied.

See all my Salt-related posts