unicolet.org

One notable thing I did in my first 25 years was in answer to an odd request I got from a customer of a customer (don't ask). This particular organization had lost control of their own authoritative DNS (public DNS) and needed a hand to recover the zone hosted there. Unfortunately this was the only DNS so taking it down to mount the disk would have resulted in unacceptable downtime and there was also the risk that the disk had been encrypted (they were not sure). So many things were unknown about this server that even a reboot was considered risky. So, what we ended up doing instead was mirroring the network traffic on the switch to a new server, run tcpdump on all DNS traffic for a couple of weeks, and then through a series of specially crafted tshark + awk commands we rebuilt the entire zone file (which was not very large, thankfully). We reviewed the zone file with the customer, loaded into a new server and then swapped it in while keeping the old system running. I never heard f...

4 podcasts to listen to while commuting (in no particular order): DevOps cafè  : hosted monthly, features great guests and packs a tremendous amount of valuable information. Warm up by listening to a great episode with with Tom Limoncelli MagPipe Talk show  : I discovered this gem because it recently featured Martin Fowler The Changelog : more targeted towards developers than the previous two. Make sure you listen to the recent episodes with Yehuda Katx and Mitchell Hashimoto Hansel Minutes . Usually this one is mostly of interest to Windows people. Useful if one wants to keep an eye on the ther side of the pond or catch the occasional high-level talk

Boot and install a new KVM guest with guest agent support with this super simple libvirt one-liner (great for scripts and CI deployments). Tested on a Centos 7.1 host creating a Centos 7.1 guest from a basic Kickstart file. Parameters should be quite self-explanatory, in any case here's the documentation for virt-install . Important : remember to change the unix socket path on the last line. File: virt-install_auto.sh -------------------------- virt-install --name "guest01" --memory 2048 -l http://your.ris.server/ris/centos71 -x "ks=http://your.ris.server/ris/ks.cfg console=ttyS0" --disk size=8,pool=your_pool,bus=virtio,format=qcow2 -w default --graphics vnc --channel unix,mode=bind,path=/var/lib/libvirt/qemu/guest01.agent,target_type=virtio,name=org.qemu.guest_agent.0

When using ZFS on top of iSCSI devices one needs to deal with the fact that iSCSI devices usually appear late in the boot process. ZFS on the other hand is loaded early and the iSCSI devices are not present at the time ZFS scans available devices for pools to import. This means that not all ZFS pools might be imported after the system has completed boot, even if the underlying devices are present and functional. A quick and dirty solution would be to run  zpool import <poolname> after boot, either manually or from cron. A better, more elegant solution is instead to hook into systemd events and trigger zpool import as soon as the devices are created.

Borrow a page out of the snowplow book and log asynchronously from any script with a GET request to a central system. In Visual Basic Script: URLGet "http://your.server.com/action/" & WshNetwork.ComputerName & "/" & activity & "/message/" & strValue Function URLGet(URL) Set Http = CreateObject("Microsoft.XMLHTTP") Http.Open "GET",URL,True Http.Send End Function And get analytics for free on top of your scripts with logstash (or snowplow).

I will show how to use ZFS instant snapshotting and cloning functionality to effortlessly clone a running postgres database regardless of its size. Setup Install your Linux OS of choice then ZFS and Postgres. I use Centos 7 but most commands used in this post are distro-indipendent. Create a zfs pool called tank or use whatever name suits you. In the pool create a filesystem called pgdata . For the sake of following a minimalist ZFS best practice apply the following settings: zfs set compression=lz4 tank/pgdata zfs set xattr=sa tank/pgdata

Everyone knows that OpenNMS is a powerful monitoring solution, but not everyone knows that since version 1.10 circa it embeds the Drools rule processing engine. Drools programs can then be used to extend the event handling logic in new and powerful ways. The following example shows how OpenNMS can be extended to detect missed executions for recurring activities like backups or scheduled jobs.

Learn more about Rundeck . Now that I have a fair number of jobs scheduled by Rundeck, how do I periodically prune the job execution history and keep only the last, say, 30 executions for each job?

How do you, at once, extend a LVM logical volume on a fleet of identical linux ( Centos ) servers using SaltStack ? Here's how and, thanks to Salt, it only took 5m.

Just a quick note to self that to enable/disable/query event log registration from the command line on Windows releases greater than XP and Server 2003 you can use the wevutil tool. For example to enable logging of print requests on Windows 7 for auditing purposes: wevtutil sl Microsoft-Windows-PrintService/Operational /e:true The equivalent command for the the above on Windows XP is the following: reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print\Providers /v EventLog /t REG_DWORD /d 7 /f net stop spooler net start spooler

Updated Oct 16 2013: shadow copies, memory settings and links for further learning. Updated Nov 15 2013: shadow copies example, samba tuning. Unless you've been living under a rock you should have by now heard many stories about how awesome ZFS is and the many ways it can help with saving your bacon . The downside is that ZFS is not available (natively) for Linux because the CDDL license under which it is released is incompatible with the GPL. Assuming you are not interested in converting to one of the many Illumos distributions or FreeBSD this guide might serve you as a starting point if you are attracted  by ZFS features but are reluctant to try it out on production systems. Basically in this post I note down both the tought process and the actual commands for implementing a fileserver for a small office. The fileserver will run as a virtual machine in a large ESXi host and use ZFS as the filesystem for shared data.

A common problem with OpenNMS is how to monitor a log file and trigger alerts when certain conditions are met. Let me clarify with an example: you have this mission critical app that sometimes experiences internal errors. The application keeps running and still responds to requests, but the error will slow down the system and/or delay further processing. Monitoring the process and/or network polling will obviously not be able to detect the issue and the only way is to tail the application log file and look for certain messages. The problem can usually be solved simply by forwarding the log file to OpenNMS through syslog, but what for logs generated by applications that don't speak syslog or if you don't want to configure syslog forwarding?

In enterprise environments VMware vSphere is the most common virtualization choice. Hence when it comes to provisioning most rely on vm templates to accelerate and standardize the creation of new virtual machines. While templates work well for both Windows and Linux vms it still leaves a lot out and administrators may have additional work that has to be carried out manually (like configure networking, optionally move the vm in the right network/resource group, create application accounts, set up IIS, tomcat, java, etc, etc). True, most of this post-bootstrap setup can be taken care of with Puppet, Chef or, in my case, Saltstack , but what I wanted was a push-button vm provisioning that does all of the following automagically: create a new vm (only Linux vms are supported as of now) assign a pre-determined static ip address/name move vm into appropriate network configure it with Saltstack register the vm into out monitoring system (currently OpenNMS )

Disclaimer: this is just a shameless post to get myself a place at the great Rome event ;-). Oh well, this does not mean this post is not interesting to read. When I went to the Extreme Programming conference in Alghero (Sardinia) in 2001 I was consulting mostly as a Systems Administrator. So I felt a little bit like a fish out of water and actually one of the participants asked me: do you think there are aspects of XP that can be applied to systems administration? I think I said yes, but at that time it was kind of hard for me to find points of contact between the two. Maybe unit testing could be associated with putting a pervasive monitoring in place so that when I refactored a configuration I would know if it worked before clients did. Or coding standards could be associated with using automated installers for deploying servers, but what about keeping the configuration in sync after, when the systems went into production? And what about the rest of the rules? Last bu...

See all my Salt-related posts Welcome back! In our quest to simplify the configuration and automate our systems we have  installed  Salt on all our servers and then moved on to some  basic state management . We want of course to do more sophisticated stuff with salt and we'll get to that too. But first we want to make sure that all minions are aligned to the same salt version (the latest in this case). To do that we will add another state to our configuration which we will call (very much unimaginatively) salt.sls. The content is below: salt-minion: pkg: - latest service: - running - watch: - pkg: salt-minion This instructs minions to upgrade the salt-minion package on the node and, if upgraded, restart the service. To activate this state we'll edit the top.sls state file as follows: base: '*': - ntp - salt We are now ready to apply the changes. Let's start with a guinea-pig minion: [prompt]# salt 'expendab...

Welcome to the third episode of the series ! In the previous posts we have installed salt on CentOS machines and then moved on with a basic state configuration (we will cover more in the coming postst). Now it's time to handle those pesky SLES hosts for which there are no pre built binaries. Therefore we'll have to install salt using pip . I'll cover SLES11 in this post as that's the only variant I have. Hopefully other versions should require only minor changes. Note: active subscription to Novell update service is required as the following packages can only be found on SLES 11 SDK (it's an iso, and a large one, so if you don't have it around start downloading it before you start):  python-devel libopenssl-devel zlib-devel swig Installation Add the SDK iso in the Software Management sources. Then, as root, run the following commands (answer yes when required): zypper in gcc-c++ python-devel libopenssl-devel zlib-devel swig zypper -p http://downlo...

After part 1  of this series I had Salt running properly on all minions. It's now time to get some work done with it. We will start with something simple like making sure that ntp is installed and running on all minions. In order to to do that we will use the Salt  states enforcement feature. The default salt states configuration requires that: state definitions be kept in  /srv/salt the default state be named top.sls We will probably need to create both the directory and the files, which we can do with the following command (check that you are not overwriting your own state, needs to be done on the master only!): mkdir -p /srv/salt cat <<EOF >/srv/salt/top.sls base: '*': - ntp EOF What this state definition means is that the base state requires all nodes (as selected by '*') to apply the ntp state. Since we have not yet defined an ntp state we are going to do it right away: cat <<EOF >/srv/salt/ntp.sls ntp: pkg: - ins...

This post is the first in a series documenting the deployment of Salt on a small network ( ~ 100 hosts, initially targeting only linux-based ones which account for roughly half of it). Due to the low number of hosts I have gone for a single master layout. The linux hosts are for the greatest part running Centos 5.[4,5] in both x86 and x64 favors, and just a couple running SLES. Installing salt master The easiest way to install salt on Centos is to pull in the epel repository  : rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm then install salt with yum: yum install -y salt-master Since minions by default will attempt to connect to the salt master by resolving an host named  salt  I configured a salt cname record for the salt master host in the dns server. At this point the master can be started with: /etc/init.d/salt-master start Note : I don't have firewall or SELinux enabled. In particular SE...

When developing applications for the cloud everybody knows (or should know) that a host, network or disk (in short any resource) failure is not an exceptional event but a rather common one. A resource failure becomes a 'normal', common event in the application lifecycle like the occasional bug. The Amazon approach is that some services (like databases) come with a certain degree of resiliency built-in while others (i.e. EC2 instances) are expected to fail relatively frequently and it is left to the developer to install backup, redundancy and availability countermeasures. My understanding is that other providers, like Rackspace , have instead a more traditional approach and will automatically restart failed virtual servers in case of host failure. If the failed cloud server image cannot be recovered then it will be bootstrapped from the most recent backup. This means that, depending on the requirements, one could move a traditional application to the cloud without having to...

QNAP devices have snmp support out-of-the-box, unfortunately the agent they ship with is almost unusable. At first it seems it supports lots of cool features (like fans, temperature, smart, etc) but if you take a little time to dig deeper you will notice that almost all key entries are, what?!, octetString s. Capacity reported as a String, QNAP what were you thinking? So good luck estimating disk usage when it is reported as a string: '1.8TB'  (I quote exactly as it is shown by mibbrowser) . Without wasting any more of your (and mine) time let's fix that by installing the Optware QPKG and then installing net-snmp from the ipkg web console. The whole process is traightforward, just make sure to have a recent firmware: log in to the QNAP admin web interface open Applications servers and then select QPKG Center from the Available tab install Optware and the enable it from the Installed tab now access the Optware web interface and search for net-snmp, then click th...