Make Your Monitoring Speak

Justin Seitz

Sys admins eagerly anticipate the smug email we are going to send when we fix an overnight problem; there is nothing more satisfying. In contrast, we also dread the days when we arrive at the office and discover that the mail server has been down for 12 hours, thus not sending notifications nor allowing the marketing team to close that big deal. It is in these times that we always want more from our monitoring.

In this article, I am going to explore how to get Nagios (http://www.nagios.org/), a popular open source monitoring solution, and the Asterisk PBX (http://www.asterisk.org/) to work together in harmony. I'm going to be using a few of the extended features in Asterisk, as well as a text-to-speech synthesis engine called Festival. By having your monitoring software place direct voice calls to yourself and other systems administrators in your organization, you have an alternative to email, SMS, pagers, and instant messengers.

For the purpose of this article, I am going to assume that you have a working install of Nagios and an operational Asterisk PBX. PHP must also be installed on both of these machines. The Festival speech synthesis system and a software phone for testing are also required; in my example, I use XLite. I am basing all of these examples on a CentOS 4.3 install, but it should be *nix agnostic.

Monitoring System Health with Nagios

Nagios is a network and system monitoring solution that's freely available for any *nix variant. Nagios runs as a daemon on your monitoring system, with the option of having plug-ins run remotely on the monitored system to return more detailed information from the machine. It is extremely extensible in that you can define external scripts that are to be called when a service outage is detected. We are going to be working extensively with this functionality.

If you don't have a current install of Nagios, you can acquire the packages and quickly do the install. If this is your first time installing Nagios, then I recommend using the make install-config command after your final compilation. This will give you handy sample configuration files that will get you up and running quickly. The Nagios mailing lists are well used and you can usually get quick answers from the group if you are stuck.

You are going to quickly create a custom service definition that is nothing more than a ping check on a remote machine, and you are going to point the notification to your custom script that is going to contact your Asterisk machine. See Listing 1. That's it for Nagios. As soon as a check-host-alive check fails, you want your voice call to be placed.

Asterisk Manager Configuration

In order for your Nagios machine to interface to Asterisk, you must connect to the Asterisk system and execute commands on it. This is done via the Asterisk Manager, which allows an administrator to flexibly use external software to make the PBX execute certain actions.

To enable the Asterisk manager, you must edit the manager.conf file in your default Asterisk configuration directory (mine is in /etc/asterisk/). You are going to enforce some basic security policies so that only your Nagios box can connect. See Listing 2.

Next, enable the Asterisk manager on port 5038 and bind to all IP addresses. The displayconnects value will allow you to watch the Asterisk CLI for your connections as they come in. You can turn this off after your debugging is complete.

The second portion enables a user named nagios from IP address of 192.168.10.10 and gives that user an associated password with which to authenticate. These values will be necessary to use in your Nagios script. You must reload Asterisk for the changes to take effect.

Asterisk Dialplan Configuration

The first step is to give your script access to a dedicated extension that will fire your second script to do the text to speech conversion. Edit the extensions.conf file in your default Asterisk configuration folder. See Listing 3.

Next, set up a test phone to ensure that the voice calls go through. You also must create a new extension for your Nagios script to call. This extension will then be instructed to use a script on the Asterisk machine to do the final text-to-speech translation and play the audio to the connected phone.

In my test scenario, I used the X-Lite softphone (http://www.xten.com/), which you can download for Windows, Linux, and Mac. To set up a quick extension for this phone, edit your sip.conf file in the default Asterisk configuration directory (Listing 4).

Writing the Nagios Script

You must first write the script that your Nagios machine calls when the check-host-alive service detects a downed host. This script is going to connect to the Asterisk manager and save the hostname and IP address values to the Asterisk DB for use in your text-to-speech voicecall. See Listing 5. Save this script on your Nagios machine as /usr/local/nagios/libexec/callphone.php.

In Listing 5, the script first takes the command-line arguments in from Nagios and connects to Asterisk. Next, it stores the hostname and IP address in the Aseterisk database and originates a call on your softphone to the extension 555. As set up in your dialplan, you know that any calls to extension 555 will be rerouted to our asterisk AGI script.

Asterisk AGI Script

The Asterisk Gateway Interface is a way to add additional functionality to your existing Asterisk install. You can make Asterisk do a myriad of different things using AGI scripts and in this example case, it is going to take the originated call from your Nagios script, read out the database values you set, do the text-to-speech conversion with Festival, and finally speak your notification over the phone.

In my case, I didn't install Festival with Asterisk, so I did a quick:

yum install festival 
            

http://www.cstr.ed.ac.uk/projects/festival/ 

That's it! This script just takes the variables out of the Asterisk database and alerts you to your downed host. Furthermore, if you have enabled logging, it will also log everything to file for debugging purposes.

Testing the Whole Operation

After you have your phone correctly connected to your Asterisk machine and you are positive you can make calls and receive them, you must ensure your monitoring will work correctly with Asterisk.

To do a quick test, I set up a dummy sub-interface on one of my internal test machines by doing the following:

ifconfig eth0:1 inet 192.168.11.11 netmask 255.255.255.0 \
  broadcast 192.168.11.255 

ifconfig eth0:1 up  

Using some extended features in Nagios, you can make this the primary method of contact or an escalated contact method. You can also have it call groups of people at the same time, broadcast phone calls, and many other options all documented in the Asterisk configuration manual.

Conclusion

As your organization and resulting network grows, so will your need to ensure that your infrastructure stays alive and well. In this article, I have illustrated a unique way to keep all of your systems administration staff in the loop as these changes occur.

Spend some time playing with your setup, changing the voices that call out, creating your own custom messages, or whatever your heart desires. For further information, refer to the References section below, and remember to tell all your friends that your monitoring now talks to you.

References

Asterisk PBX Homepage -- http://www.asterisk.org/

Festival Speech Synthesis System Homepage -- http://www.cstr.ed.ac.uk/projects/festival/

Nagios Exchange -- http://www.nagiosexchange.org/. This is a place where people submit custom Nagios plug-ins and add-ons and is a valuable resource for finding out the bleeding edge in monitoring.

Nagios Homepage -- http://www.nagios.org/

Nagios Documentation Page -- http://www.nagios.org/docs/

Voip-Info.org -- http://www.voip-info.org. This is an excellent resource for all things VOIP. It has a thorough explanation of all the configuration settings in Asterisk, as well as some coding examples.

Justin Seitz is a freelance technical writer and systems administrator. When not writing articles or pounding away on his console, he's on the beach with his lovely wife and two children. Special thanks to Ken Hess for all the help and guidance in getting this article out. Justin can be reached at Justin.seitz@gmail.com.