unicolet.org

Everyone knows that OpenNMS is a powerful monitoring solution, but not everyone knows that since version 1.10 circa it embeds the Drools rule processing engine. Drools programs can then be used to extend the event handling logic in new and powerful ways. The following example shows how OpenNMS can be extended to detect missed executions for recurring activities like backups or scheduled jobs.

One of the nice aspects of OpenNMS is that, out of the box, it will collect a lot of data from most snmp-enabled resources. The downside is that such collection is I/O heavy (iops, not throughput). Even on moderate installations with hundreds of nodes it is enough to swamp even the fastest disk subsystem (except for those with controllers supported by large write caches). A symptom is that I/O wait will be quite high on the opennms box itself. I/O Wait before and after switch jrobin backend from FILE to MNIO

OpenNMS 15 puts a much higher load on the database than previous versions. Besides tuning postgres, the OS and perhaps splitting the app and the db on different boxes one aspect that I found to really make a difference is having a warm postgres cache.

Disclaimer: Packt kindly sent me a free copy for review. TL;DR: Rating 4/5. Recommended for beginners and intermediate. The book itself is short, but packed with information. A fast reader with some experience with OpenNMS should be able to finish it in 4 to 6 hours. Beginners will probably want to follow the pointers to the online documentation, check the configuration files and possibly experiment so they should allocate more time. Before being published the book has been reviewed by Jeff Gehlbach. Anyone who has been involved with OpenNMS for some time know him, as he is one of the many brilliant minds working for the OpenNMS company, the commercial entity which develops and supports OpenNMS. Surely his involvement serves as a kind of seal of quality for the book. I for one was surprised by the clarity with even the most complex aspects of OpenNMS were presented in such a short text. Instant OpenNMS Starter is divided in three main parts: installation, quick start...

Going beyond the normal application availability check One interesting use of the  OpenNMS JDBC poller is for extracting data from the Oracle administrative database tables, for example tracking tablespace quota usage to detect quota exhaustion, sudden usage peaks and graph usage over time. Graph of quota usage for user [redacted] on tablespace DAT. Notice the cleaning operation running at 3.30 AM Tablespace quotas is a feature present in the Oracle database that allows the DBA to set a limit on the amount of storage that any given user can consume on a specific tablespace. This allows the DBA to share tablespaces across users yet still be able to policy users into predefined usage boundaries. When a user consumes all its quota it can no longer store data, but it can delete it, thus allowing self-recovery.

As a long time OpenNMS user I've been often impressed with its extensibility and the completeness of its feature set. There is support for lots of data collection techniques: from the old school snmp exec extensions, to the http poller, from the JDBC poller to the XML poller and many others  that I probably forgot to mention. Supporting new probes is therefore just a matter of how,  not  if  , it can be done. And with new monitoring tools popping up every day this is clearly good as it allows OpenNMS to keep up with the competition. So the present looks bright, but what about the future? With OpenNMS 2.0 not yet on the radar I thought I could put together a list of features I would love to have. What do you think of them?

A common problem with OpenNMS is how to monitor a log file and trigger alerts when certain conditions are met. Let me clarify with an example: you have this mission critical app that sometimes experiences internal errors. The application keeps running and still responds to requests, but the error will slow down the system and/or delay further processing. Monitoring the process and/or network polling will obviously not be able to detect the issue and the only way is to tail the application log file and look for certain messages. The problem can usually be solved simply by forwarding the log file to OpenNMS through syslog, but what for logs generated by applications that don't speak syslog or if you don't want to configure syslog forwarding?

In enterprise environments VMware vSphere is the most common virtualization choice. Hence when it comes to provisioning most rely on vm templates to accelerate and standardize the creation of new virtual machines. While templates work well for both Windows and Linux vms it still leaves a lot out and administrators may have additional work that has to be carried out manually (like configure networking, optionally move the vm in the right network/resource group, create application accounts, set up IIS, tomcat, java, etc, etc). True, most of this post-bootstrap setup can be taken care of with Puppet, Chef or, in my case, Saltstack , but what I wanted was a push-button vm provisioning that does all of the following automagically: create a new vm (only Linux vms are supported as of now) assign a pre-determined static ip address/name move vm into appropriate network configure it with Saltstack register the vm into out monitoring system (currently OpenNMS )

Disclaimer: this is just a shameless post to get myself a place at the great Rome event ;-). Oh well, this does not mean this post is not interesting to read. When I went to the Extreme Programming conference in Alghero (Sardinia) in 2001 I was consulting mostly as a Systems Administrator. So I felt a little bit like a fish out of water and actually one of the participants asked me: do you think there are aspects of XP that can be applied to systems administration? I think I said yes, but at that time it was kind of hard for me to find points of contact between the two. Maybe unit testing could be associated with putting a pervasive monitoring in place so that when I refactored a configuration I would know if it worked before clients did. Or coding standards could be associated with using automated installers for deploying servers, but what about keeping the configuration in sync after, when the systems went into production? And what about the rest of the rules? Last bu...

QNAP devices have snmp support out-of-the-box, unfortunately the agent they ship with is almost unusable. At first it seems it supports lots of cool features (like fans, temperature, smart, etc) but if you take a little time to dig deeper you will notice that almost all key entries are, what?!, octetString s. Capacity reported as a String, QNAP what were you thinking? So good luck estimating disk usage when it is reported as a string: '1.8TB'  (I quote exactly as it is shown by mibbrowser) . Without wasting any more of your (and mine) time let's fix that by installing the Optware QPKG and then installing net-snmp from the ipkg web console. The whole process is traightforward, just make sure to have a recent firmware: log in to the QNAP admin web interface open Applications servers and then select QPKG Center from the Available tab install Optware and the enable it from the Installed tab now access the Optware web interface and search for net-snmp, then click th...

I have just completed an upgrade from OpenNMS 1.8.11 to the latest and greatest 1.10. The upgrade in itself is easy and the guides on the OpenNMS wiki will serve you well. Instead in this post I'll describe a couple of other changes that I made which improved very much the overall performance and responsiveness of the system. One is the upgrade from PostgreSQL  8.4 (which came with CentOS) to 9.1 + tuning. The other is switching from apache to nginx. Upgrading postgres is mostly a matter of taking a backup,  pulling in the right repo , running  yum install  and finally importing the database. Tuning postgres I left opennms running on PostgreSQL 9.1 for a while and then I went checking how well postgres was doing. Postgres 9 already performs significantly better that its 8.x predecessors, but I wanted to do better than out-of-the-box. As the postgres user I logged in into the opennms database to install a utility that will help me estimate how muc...

I have an OpenNMS server (a PC really) sitting under a desk at a company that I don't hear from often, but I still get weekly backup mail-recaps. This week the recap brought bad news: postgres stopped accepting connections until the db is vacuumed to prevent transaction id wraparound. I have know of this issue of postgres for some time and planned accordingly: a cron job every month would run a full vacuum. Unfortunately in one month the db grew so much that vacuum never reached completion because the customer simply rebooted the server thinking that it was 'stuck'. This vacuum-stuck-reboot had gone on for months until today. As I knew from prior experience this was going to be a painful experience: a quick du on the database folder reported 60GB of data. Vacuum would probably take days. Luckily for me I had heard from the great Postgresql 9.0 high performance book that there is another tool in the toolbox called cluster . So I decided to execute this plan this ...