Automating Signature Updates for Cisco IPS/IDS Sensors

Lisa Hamet Bernard

As the variety, sophistication, and sheer volume of server and network threats increase, so does the demand for Intrusion Prevention Systems/Intrusion Detection Systems (IPS/IDS). These network devices recognize malicious traffic, including viruses, worms, and various traffic patterns indicative of hacking techniques targeting both operating systems and applications.

The network filtering to determine the presence of such events is based upon a set of " signatures" , packet sequences that define each intrusion. When an event is detected, an alert is triggered, and in the case of IPS devices, traffic from the offending IP address is immediately blocked. But, like anti-virus software on PCs, IPS/IDS devices are only as effective as the latest signatures of which they are aware. Security software companies rush to fingerprint new threats as soon as they are discovered and release signature updates that can detect these threats. Systems and network administrators must be just as proactive by installing these updates as soon as they are available.

Cisco Systems, Inc. offers a family of IPS/IDS sensors -- both standalone appliances and switch/router modules. Cisco releases regular signature update files as new threats are discovered, which can vary in frequency from daily to every few weeks. Updates are made available on Cisco's FTP site and announced via a mailing list to which anyone with a valid CCO (Cisco Connection Online) account may subscribe. The operating system includes an automatic upgrade utility feature that installs an update from a local file server on a configurable schedule. However, automating signature downloads to the local file server requires purchasing either Cisco Security Manager (CSM) or its predecessor, CiscoWorks VPN/Security Management System (VMS). Both products are costly and are Windows based (although VMS, which is being phased out, offers a Solaris version as well).

Without management software, administrators supporting these sensors must manually retrieve signature updates. I support a small network for one of my customers, for which purchasing this software was not an option. So I developed my own Perl scripts that run on a Solaris box to (1) automate the update discovery and retrieval task, and (2) verify success and send an email notification following the actual update installation. In this article, I will describe the details of these processes, highlighting remote management of a Cisco IPS device via SSH and explaining the integration with the IPS automatic upgrade feature.

The Task at Hand

My customer's IPS solution consists of two Cisco Adaptive Security Appliance (ASA) 5510s, configured (in failover mode) to provide network VPN access. Each ASA has an Advanced Inspection and Prevention Security Service Module (AIP-SSM) installed, which serves as an IPS filter on the VPN traffic. Note that although the ASAs are configured in active/standby failover mode (i.e., only one is ever active), the AIP-SSMs are both always active. Thus, neither one has any awareness of the active state of the ASA in which it is installed nor of the other AIP-SSM. So, while the ASAs automatically synchronize any configuration changes made on the active one, the AIP-SSMs must be managed independently. The AIP-SSMs are running IPS version 5.1(4) at the time of this writing. Since an equivalent operating system image also runs on the rest of the Cisco family of IPS/IDS Sensors (IPS-42xx and IDS-42xx appliances, IDS Network module for 26xx, 3660 and 37xx Router families, and the Catalyst 6500 IDS Module 2),

I will refer to my targets as IPSes. Though not tested on any of these other platforms, my scripts should work for any of these sensors as well.

Automating the IPS signature update process includes the following steps:

1. Determine the signature version currently installed on the IPSes.

2. Determine the latest signature version available on Cisco's FTP site. If it is newer than the installed version, download it to a local server.

3. Send email to the admins that an update was downloaded with timestamp and new signature version.

4. Check the Event Store on the IPSes to verify that the update was installed and, if so, send email to the admins with the details.

Of course, you must also configure the auto-upgrade-option on the IPSes. It is best to schedule the auto-updates to occur shortly after the signature download script runs.

Remote Connectivity to the IPSes

Both scripts (getupdate and getipsstatus) run on a Solaris 8 server under Perl v5.8.3. The tricky part of my code development was successfully establishing and sustaining a remote session with the IPSes via Perl. Using telnet was not an option; it is against my customer's acceptable use policies to use telnet, and it is even disabled by default in the IPS operating system because of its inherent security issues. So SSH was the choice. The Solaris 8 server is running OpenSSH_4.0p1 (SSH protocol version 1.99).

SSH version 1 is currently universally supported by Cisco; however, the exact implementation varies based on the underlying operating system of the particular device. (IPS version 5.1(4) includes OpenSSH_3.7.1p2.) Because of this, there is not one magic Perl module/method combination that works across all Cisco devices, and it became an exercise of trial and error to find the right combination for the IPSes. After much experimenting, including attempts with both Net::Telnet::Cisco and Net::Appliance::Session, I found success using a combination of Net::Telnet (version 3.03) and Proc::Spawn (version 1.03). The details will be described later.

I used preshared keys for SSH authentication from the Solaris server to the IPSes so system passwords would not have to be stored in my script, especially since the IPS account used must have administrator privilege. I created an account named sigupdate on both sides. Then I generated SSH RSA version 1 keys for user sigupdate on the Solaris server and added its public key to both IPSes as an authorized public key. Now the scripting can begin.

Establishing an SSH Session

The getupdate script is the first piece of the process (see Listing 1). The first step in this script is to determine the currently installed signature update version on the IPSes. I use the spawn_pty function from Proc::Spawn to spawn an SSH process as user sigupdate to one of the IPSes. (I chose the second IPS, since it is usually idle, installed in the ASA that is normally in standby mode.) Note the " 1" parameter to SSH. I must force the SSH client on the Solaris server to negotiate a version 1 only session, because the IPS accepts only RSA version 1 keys. Without this parameter, the Solaris side only attempts a version 2 connection/authentication and fails.

The spawn_pty function returns a process ID and an IO::Handle object to use for all I/O. Next, I create a new Net::Telnet object named ssh to read from and write to this already open handle. The Prompt parameter defines the prompt that the remote side of the SSH connection will supply. This is key, because the Net::Telnet method cmd reads the input stream looking for a match on this string to know that the command has completed. If the pattern is not matched, "cmd" will fail with a time-out error. The Telnetmode parameter turns off the feature to look for specific telnet session control character sequences since the session is SSH, not telnet. The Cmd_remove_mode parameter removes the echo of the sent command from the output sent back by the IPS. Finally, the Output_record_separator parameter defines the output line delimiter for the cmd method. Though no longer included, I used the Dump_log and Input_log parameters to capture session traffic to a file during the development process.

Once the SSH session handshake has been initiated, I call the waitfor method to read the input stream looking for the prompt to know that the session is established.

Identifying the Currently Installed Signature Version

I need only two commands to determine the currently installed signature version, and both are sent to the IPS using the Net::Telnet cmd method. First, since this is not an interactive session, I disable output paging with terminal length 0 Second, I issue a show version, and capture the output into the array lines. I parse lines to find the line containing " Signature Update" ; then I parse that line to grab the version number. It will be an " S" followed by a three-digit number. I capture the three-digit number as variable currentver.

I close the SSH session with the Net::Telnet close method. I found that this method does not kill the SSH process, so I also issue a system call to kill the process ID.

Determining the Latest Available Signature Version

For this part of the getupdate script, I use the Net::FTP module (version 2.75). The Cisco FTP site, ftp-sj.cisco.com, only accepts passive FTP connections, so the parameter Passive must be specified when creating the new Net::FTP object (called ftp). A valid CCO account username and password is the login for this site. Because the password must be hard-coded into this script, remember to update it when you change it on your CCO account. The next step is to use the Net::FTP cwd method to change to the appropriate directory.

The IPSes I manage are running release 5.1(4), so my script goes to the 5.x/sigup directory. Next, I use the ls method to get a directory listing and store it in the array lines. I parse each line of this array searching for the signature update number of the referenced file. There are two files associated with each update: the update itself and a README file. The README filename does not precede the version number with an " S" , so I check whether the version is newer (i.e., greater) than currentver and includes the " S" . If I find a newer version, I use the Net::FTP binary method to specify a binary transfer, then call the get method to download this signature update file into an archive directory on the Solaris server.

The archive directory must be writeable by user sigupdate, because this script must run as user sigupdate for the appropriate RSA keys to be exchanged during the SSH session establishment. This is the location from which the update will be downloaded and installed into the IPSes using their auto-upgrade-option. Finally, I quit the FTP session and issue a notification email to an administrator address with the newly downloaded signature update version number and timestamp of the download.

Configuring the IPSes

With the update available, the IPSes can be set to check the archive directory on the Solaris server on either a periodic or a calendar schedule. I configured a periodic schedule, specifying the upgrades to occur at 4:30 and 4:40 a.m., respectively, and every 24 hours thereafter. The command used to specify this feature is " auto-upgrade-option enabled" , configurable in service host submode if you are using the command-line interface. The other necessary parameters to set when enabling this feature all relate to the local file server containing the update: IP address, directory in which the upgrade files are located, and username and password with which to connect. (All of these values are referenced earlier in this article, using the Solaris 8 server.) Lastly, the file copy protocol to use must be specified, either FTP or SCP. As with telnet, my customer does not allow FTP for security reasons, so I use SCP.

Unfortunately, it is not possible to use pre-shared keys for authentication in this direction (from the IPS to a server). This is because there is no way to generate an RSA key pair for a specific user account on the IPS, so there is no public key to add as an authorized key on the Solaris server. This limitation is magnified by the fact that the relevant password is printed in clear text in the output of a " show configuration" command on the IPSes. (I have submitted this issue to Cisco, and it has been logged as a bug to be addressed in future releases.) Therefore, best practice dictates limiting access and privileges as much as possible for the sigupdate account on the Solaris server. Note that, in addition to setting the file server's IP address when configuring auto-upgrade-option, you must also add it to the IPS's SSH known hosts list in order to use SCP. (Enter the IP address with the " ssh host-key" command, and the IPS will automatically contact the server over the network to obtain its public key.) The auto-upgrade-option is now ready to go.

Monitoring Upgrade Results

The last piece of the process is to verify that the update was successfully installed on each IPS and email the admins with the details. This is accomplished with the getipsstatus script (see Listing 2). The IPS logs the sequence of steps performed during the upgrade in its Event Store. An example of the final Event Store entry for a signature update installation can be seen in Figure 1.

The getipsstatus script searches the Event Store on each IPS for this entry. Each IPS to check is listed in an external file (I use ipslist.txt), in the format of IPSname and IP address, separated by a space, one per line. For each IPS listed, the procedure is as follows.

To begin, the IPS name and IP address are read into corresponding variables, and the prompt string for the IPS is created using the name. Because I will again be accessing the IPSes via SSH, the prompt is necessary for the Net::Telnet cmd method to complete correctly. Then, I establish an SSH session, using the same procedure as in getupdate. After disabling paging on the IPS, I issue the IPS command show events past 00:30:00 | begin softwareUpgradeCompleted. The " past 00:30:00" qualifier directs the IPS to go back 30 minutes in the Event Store, because I intend to execute this script shortly after the automatic update process is scheduled to run.

The | begin softwareUpgradeCompleted filters the output to start with the first event entry that includes this string. The show events command behaves like the shell tail -f command (i.e., an open pipe from the Event Store), so it requires further input to complete. Therefore, I must use the Net::Telnet print method combined with the waitfor method instead of the cmd method. The print method sends the command, and the waitfor method captures the input stream into the eventmsg array until it matches on the sequence " CR-CR-LF-CR-CR" that separates event entries in the command output. If I find this string, I have matched an upgrade entry, so I call the print method to send a Control-C to end the show events command.

Finally, I close the SSH session as I did in getupdate and email the administrator account with the contents of eventmsg. Alternatively, if the entry is not in the Event Store, then the waitfor method errs on time out. I define the waitfor parameter Errmode as an array named Notfound, which is the action to perform on error. The first element of Notfound is the subroutine to call (named noupgrade), and the other elements are the arguments to noupgrade. The noupgrade subroutine kills the SSH process, forces the script to continue to the next entry in ipslist.txt, and optionally sends email to the administrator account that an upgrade was not performed on the given IPS.

Setting Up the cron Jobs

The getupdate script includes the intelligence to determine whether there is a new update available, so it can run as a regular cron job without active monitoring for Cisco's update availability notification email. I have it set to run at 4:00 a.m. daily, just before the automatic update schedule on the IPSes.

Updates have not been released more often than once a day, so a higher frequency seems unnecessary. More evidence for this scheduling: In Cisco's IPS management software, the feature for Automatic Signature Download allows entry of one time setting (per 24-hour period) at which new signature availability is checked.

I set the getipsstatus script to run at 4:55 a.m. daily, within 30 minutes of the possible update (as coordinated with the show events command in the script). It is called with the name of the file containing the list of IPSes. Figure 2 illustrates the crontab entries for these scripts. Note that both scripts must run as user sigupdate so that the correct SSH pre-shared keys are exchanged.

Final Notes

Thanks go to Oliver Gorwits at the University of Oxford (oliver.gorwits@oucs.ox.ac.uk), author of Net::Appliance::Session and other Perl modules. Oliver provided great advice based on his code development for debugging the SSH handshake process to some Cisco devices.

On the topic of future work, another feature touted by Cisco in their IPS management software is the automated collection of events logged on the various IPSes and report generation from that data. It would be straightforward to duplicate this capability as well, simply by modifying the getipsstatus script to capture many different types of events in the Event Store. This data could then be fed into your sorting/reporting/monitoring tool of choice.

Conclusion

IPS devices have become key components in maintaining a secure local area network, and every new network threat becomes a race between malicious code propagation and development and installation of signature recognition code to defend against it. I have demonstrated how to keep the Cisco IPS known signatures current with Cisco releases, without investing in management software to handle the task. Depending on the relative timing of your cron jobs and IPS automatic update running and a new signature update release, your update may occur before you receive Cisco's bulletin that it exists. Now that's proactive!

Lisa Hamet Bernard, SCSA, CCNA, is currently an independent consultant with more than 15 years experience in Unix system and LAN/WAN network administration for government and industry. She received a BSCS from the University of Maryland Baltimore County, and lives in the Baltimore-Washington, D.C. area. She loves spending time with her husband and two American Eskimo dogs. She can be reached at: lisa@lhb-consulting.com.