unicolet.org

One notable thing I did in my first 25 years was in answer to an odd request I got from a customer of a customer (don't ask). This particular organization had lost control of their own authoritative DNS (public DNS) and needed a hand to recover the zone hosted there. Unfortunately this was the only DNS so taking it down to mount the disk would have resulted in unacceptable downtime and there was also the risk that the disk had been encrypted (they were not sure). So many things were unknown about this server that even a reboot was considered risky. So, what we ended up doing instead was mirroring the network traffic on the switch to a new server, run tcpdump on all DNS traffic for a couple of weeks, and then through a series of specially crafted tshark + awk commands we rebuilt the entire zone file (which was not very large, thankfully). We reviewed the zone file with the customer, loaded into a new server and then swapped it in while keeping the old system running. I never heard f...

The ZFS filesystem has many features that once you try them you can never go back. One of the lesser known is probably the support for replicating a zfs filesystem by sending the changes over the network with zfs send/receive. Technically the filesystem changes don't even need to be sent over a network: you could as well dump them on a removable disk, then receive  from the same removable disk.

This is an attempt to record my thoughts and describe a solution with regard on how to target/classify minions in a SaltStack environment. SaltStack logo An interesting discussion on the topic can be found in this (rather old) thread on the Salt-User mailing list: https://groups.google.com/forum/#!topic/salt-users/R_jgNdYDPk0 Basically I share the same concern of the thread author Martin F. Kraft, who in an attempt to put and end to this madness ended up writing reklass . Roles seem to be easy enough to understand and provide for a clear separation between the actual infrastructure and the desired configuration state, while allowing extensibility and customization (a more specific role can override some settings from another role).

Boot and install a new KVM guest with guest agent support with this super simple libvirt one-liner (great for scripts and CI deployments). Tested on a Centos 7.1 host creating a Centos 7.1 guest from a basic Kickstart file. Parameters should be quite self-explanatory, in any case here's the documentation for virt-install . Important : remember to change the unix socket path on the last line. File: virt-install_auto.sh -------------------------- virt-install --name "guest01" --memory 2048 -l http://your.ris.server/ris/centos71 -x "ks=http://your.ris.server/ris/ks.cfg console=ttyS0" --disk size=8,pool=your_pool,bus=virtio,format=qcow2 -w default --graphics vnc --channel unix,mode=bind,path=/var/lib/libvirt/qemu/guest01.agent,target_type=virtio,name=org.qemu.guest_agent.0

When using ZFS on top of iSCSI devices one needs to deal with the fact that iSCSI devices usually appear late in the boot process. ZFS on the other hand is loaded early and the iSCSI devices are not present at the time ZFS scans available devices for pools to import. This means that not all ZFS pools might be imported after the system has completed boot, even if the underlying devices are present and functional. A quick and dirty solution would be to run  zpool import <poolname> after boot, either manually or from cron. A better, more elegant solution is instead to hook into systemd events and trigger zpool import as soon as the devices are created.

Everyone knows that OpenNMS is a powerful monitoring solution, but not everyone knows that since version 1.10 circa it embeds the Drools rule processing engine. Drools programs can then be used to extend the event handling logic in new and powerful ways. The following example shows how OpenNMS can be extended to detect missed executions for recurring activities like backups or scheduled jobs.

Learn more about Rundeck . Now that I have a fair number of jobs scheduled by Rundeck, how do I periodically prune the job execution history and keep only the last, say, 30 executions for each job?

How do you, at once, extend a LVM logical volume on a fleet of identical linux ( Centos ) servers using SaltStack ? Here's how and, thanks to Salt, it only took 5m.

A common problem with OpenNMS is how to monitor a log file and trigger alerts when certain conditions are met. Let me clarify with an example: you have this mission critical app that sometimes experiences internal errors. The application keeps running and still responds to requests, but the error will slow down the system and/or delay further processing. Monitoring the process and/or network polling will obviously not be able to detect the issue and the only way is to tail the application log file and look for certain messages. The problem can usually be solved simply by forwarding the log file to OpenNMS through syslog, but what for logs generated by applications that don't speak syslog or if you don't want to configure syslog forwarding?

After part 1  of this series I had Salt running properly on all minions. It's now time to get some work done with it. We will start with something simple like making sure that ntp is installed and running on all minions. In order to to do that we will use the Salt  states enforcement feature. The default salt states configuration requires that: state definitions be kept in  /srv/salt the default state be named top.sls We will probably need to create both the directory and the files, which we can do with the following command (check that you are not overwriting your own state, needs to be done on the master only!): mkdir -p /srv/salt cat <<EOF >/srv/salt/top.sls base: '*': - ntp EOF What this state definition means is that the base state requires all nodes (as selected by '*') to apply the ntp state. Since we have not yet defined an ntp state we are going to do it right away: cat <<EOF >/srv/salt/ntp.sls ntp: pkg: - ins...

This post is the first in a series documenting the deployment of Salt on a small network ( ~ 100 hosts, initially targeting only linux-based ones which account for roughly half of it). Due to the low number of hosts I have gone for a single master layout. The linux hosts are for the greatest part running Centos 5.[4,5] in both x86 and x64 favors, and just a couple running SLES. Installing salt master The easiest way to install salt on Centos is to pull in the epel repository  : rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm then install salt with yum: yum install -y salt-master Since minions by default will attempt to connect to the salt master by resolving an host named  salt  I configured a salt cname record for the salt master host in the dns server. At this point the master can be started with: /etc/init.d/salt-master start Note : I don't have firewall or SELinux enabled. In particular SE...