Article Listing 1 Listing 2 Listing 3 feb2006.tar

File Integrity Assessment via SSH

Hal Pomeranz

File integrity assessment (FIA) tools like Tripwire [1], Samhain [2], AIDE [3], et al. are commonly deployed in organizations to help assist forensic investigation after a security incident and as a host-based intrusion detection tool to help detect unauthorized file system changes (this also makes them useful monitoring tools for existing change control procedures, though that is not the focus of this article). The concept is simple: the administrator creates a configuration file that lists the critical system files and directories that the FIA tool should monitor, then uses the FIA tool to create a database that tracks common parameters about those files, such as permissions and ownerships, file size, and MAC times, along with one or more cryptographic checksums over the file contents (typically via common hashing algorithms like MD5, SHA-1, etc.). The FIA tool is then re-run periodically, and the current state of the file system is compared to the values stored for the various files in the database -- if there are any discrepancies, the files are flagged as having been modified and a report is generated.

The canonical problem with FIA tools, however, is protecting the database generated by the FIA tool, as well as the binary for the FIA tool itself, from unauthorized tampering by attackers who gain root access to the system. After all, if the attacker installs a rootkit and then updates the FIA database for the system to reflect the new state of the file system, then the administrator may be unaware of the attacker's changes. Similarly, the attacker could modify the FIA tool binary to either ignore or lie about the state of files installed by the attacker.

Several different strategies have been developed to try and address this problem:

  • FIA database/binary on read-only media -- One way to prevent tampering is to burn the FIA database and binary onto a CD-ROM. The difficulty with this approach is that the FIA database will need to be updated after normal system updates -- like patching and configuration file changes -- and burning a new CD after each system update becomes a huge hassle.
  • Access FIA database/binary via read-only NFS -- FIA databases and binaries for various systems could be stored on a central, secure NFS server and exported read-only to the rest of the network. Assuming you trust the security of your network, this seems like a reasonable strategy for a protected enterprise network because it provides the read-only functionality we're looking for while still allowing a relatively sane update strategy (see Jason Perlman's article in the December 2004 issue of Sys Admin for a simple implementation of this idea [4]).
  • Digital signature -- The commercial version of Tripwire protects its FIA database by validating the digital signature of the database before each use. One could imagine implementing a similar strategy with an Open Source FIA tool and PGP.

Lately, however, I've been experimenting with a different approach. The concept is to store FIA databases and binaries for the various systems being monitored on a central, highly secure system on a protected network -- call this the FIA management server. Periodically from cron, we run a script on the FIA management server that uses scp to copy the FIA database and binary to a given system and then runs the FIA tool on that remote system via SSH. Any report output and/or updates to the FIA database can then be pulled back to the FIA management server via scp, and all traces of the FIA tool can then be removed from the remote system.

Thus, the FIA tool binary and database are only available to the attacker at the instant the scan is being performed. Furthermore, there is no cron job that can be subverted by the attacker, or any other indication on the individual systems themselves that the site is even using FIA tools to monitor their systems. And as we'll see later, there are some additional tricks we can use to further deceive attackers who may be monitoring the system closely at the time the FIA scan is being performed.

One of the issues with this approach, however, is that the FIA tool must generally run with root privileges on the system being scanned. So we're assuming here that the FIA management server has remote root access via SSH to all of the systems on the network where the FIA tool is being used. This means that the FIA management server must be protected at all costs, since anybody with root access on the FIA management server will effectively have root privileges on the entire network.

Initial Setup

On the FIA management server, use the ssh-keygen command to generate a public/private key pair for the root account. Because this authentication key will be used by a fully automated cron job, you will probably want to use a null passphrase when creating the key -- note, however, that this means that the private key will be stored unencrypted in the .ssh directory in root's homedir on the FIA management server. The ssh-agent utility could be used as an alternative to an unencrypted key file, but configuring ssh-agent for this purpose is outside the scope of this article.

Once the key pair has been generated on the FIA management server, copy the public key into ~root/.ssh/authorized_keys on all of the systems you plan to monitor. Also make sure that you have set "PermitRootLogin without-password" (or "PermitRootLogin nopwd" if you're using the commercial version of SSH) in the sshd_config file on these systems, so that root logins are allowed using public-key authentication. At this point, you should verify that you can use SSH to execute commands from the FIA management server on the various remote systems without being prompted for a password or passphrase -- something like "ssh <remotehost> uptime" should be sufficient for testing purposes.

The next step is to configure the FIA tool and create the initial FIA databases for all the systems you plan to monitor. On each system, create a directory that contains the FIA tool binary and an appropriate configuration file for that system. Next, use the FIA tool to generate the initial FIA database for the machine and store that database in the same directory with the FIA tool binary and configuration file. Then, simply tar up the directory you just created, gzip it, and copy the resulting tgz file back to the FIA management server as <hostname>.tgz, where <hostname> is the name of remote system from which the tarball was copied.

The Scripts

Initially, I wrote a simple script called "check" that would run an FIA scan on a single remote system. Note that for this particular implementation I'm using AIDE as my FIA tool because it's (a) open source, (b) portable across many Unix flavors, and (c) easy to get up and running quickly. But you could easily port my scripts to use any other common FIA tool.

After using my script for a short period of time, however, it became clear that I needed a way to more easily update the FIA databases in the various host tarballs, as well as make configuration file changes when necessary -- this led to "update" and "modify-config" scripts. Because all of these scripts need to be configured with the same pathnames for the FIA install directory, database name, etc., I extracted all of this information into a single include file called "DECL" for easier maintenance.

The scripts are all fairly simple at this stage, but let's look at the check script (Listing 1) in more detail. After setting a sane PATH environment variable, the script checks its argument list. The only required argument is the name of the host to be scanned. The second argument, which is optional, is the SSH port number on the remote machine -- on many of my Internet-facing systems I run SSH on an alternate port number so I don't have to deal with constant SSH scanning and brute force password attacks on port 22. A third argument may also be added, which is the path to GNU tar on the remote system (it's easier to use GNU tar since it has built-in gzip functionality). The second argument defaults to port 22, and the third to /bin/tar (which is correct for most open source OS versions).

Next, the script declares $ROOTDIR, which is the directory where the per-host tarballs live as well as the DECL file containing other pathname declarations. After defining a temporary file for collecting script output, the check script then reads in the additional settings from the DECL file.

Let's consider the DECL file briefly (Listing 2). This file defines path and file names that will be used on the remote system being scanned:

  • REM_DIR -- The directory on the remote system where the tarball will be copied and unpacked. It's important that this directory not be one of the directories monitored by your FIA tool, because the state of this directory will change every time you run a scan.
  • REM_ARCH -- The name of the tarball (REMote ARCHive file) when copied to the remote system.
  • REM_IDIR -- The name of the directory you get when the tarball is unpacked.
  • REM_CMD -- The name of the FIA binary inside of the unpacked directory.
  • REM_CONF -- The name of the FIA config file inside the directory.
  • REM_ADB -- The FIA database in the unpacked directory.
  • REM_NEWDB -- The pathname where updated databases are created by the FIA tool running on the remote system, again relative to $REM_DIR/$REM_IDIR.

The default DECL file shown in Listing 2 has the appropriate settings to unpack the tarball as /var/spool/aide on the remote system and assumes that the directory will contain the binary aide, configuration file aide.conf, and database aide.db.

After reading all of these variable assignments from the DECL file, the check script does a quick sanity check to make sure that the tarball for the specified host exists on the FIA management server and initializes its temporary file. The main work of the script is a series of ssh and scp commands that are created using the command-line arguments to the script and the declarations in the DECL file. The output of all of these commands is captured in $TEMPFILE for later reporting.

The first scp merely copies the tarball to the remote system. Then there's a fairly complicated ssh command, but all that's really happening here is (1) a cd to the appropriate install directory, (2) unpacking the tarball, (3) another cd into the directory created by unpacking the tarball, (4) running the FIA scan, and (5) tar/gzip-ing the updated database created by the scan so that it can be copied back to the FIA management server. These last two steps will probably require some tweaking if you decide to use a tool other than AIDE.

AIDE supports the --update option, which does a normal scan and simultaneously produces a new database that reflects the current state of the system, thus making it easy to update your FIA database once you've verified that all changes in the report are "expected" (the path where the new database gets dumped is defined in your aide.conf file, and must also match the $REM_NEWDB variable in the DECL file). However, if your FIA tool doesn't support similar functionality (or uses a different command-line option than --update), you'll need to come up with another mechanism for running a scan and generating a new database and modify the script.

The next scp command in the script simply grabs the new tarball containing the updated FIA database created in the previous step and copies it back to the FIA management server and saves it to a file named <hostname>.tgz-update. Finally, the script uses ssh to remove the installation tarball and installation directory from the remote system, completely eliminating all traces of the FIA scan.

The only remaining item is to see whether there were any changes to the file system. If there were no changes, the AIDE output will contain a line of text that reads, "### All files match AIDE database. Looks okay!" If the script doesn't detect this line of output, then we're simply going to send the contents of $TEMPFILE to the standard output. Since we assume the check script is being run from cron, this should cause the script output to automatically be collected and emailed to the user the cron job is running as (typically root).

Actually, inside of the if statement we're going to use awk to filter out some of the uninteresting command output from $TEMPFILE -- specifically the output from scp that simply shows the file name being copied, as well as the warning banners emitted by the SSH daemon on the remote machine (which in my world begin "Authorized uses only..."). After that, it's just a matter of cleaning up $TEMPFILE.

The update and modify-config scripts are considerably simpler but follow the same basic pattern as the check script, so there's not much point in reviewing them here. All of the scripts, along with a sample DECL file, as well as some sample aide.conf files can be found at my Web site [5].

Some Thoughts on Obfuscation

One issue is that an attacker who happens to be monitoring the system at the time the scan is being run is likely to notice the scan happening. If they run ps, they'll see the process in the process table, and having a directory on the system called /var/spool/aide is also likely to be a real tip-off.

But it turns out that there's no need to use pathnames like /var/spool/aide, or file names like aide, aide.conf, and aide.db. We can simply choose some arbitrary but innocuous names to try to obscure the scan from a wary attacker. This is another reason why having all of the path names abstracted into the DECL file is useful -- we can easily change the names to suit our tastes, at least as long as we remember to update the file names in the host tarballs to reflect our new naming scheme.

Listing 3 shows a sample DECL file with a more generic set of file names. Using this DECL file, the AIDE directory on the remote system will end up being /var/spool/config, and the command line to run the AIDE scan will just be ./configure --config=./system.conf --update. Not much to tip off our worthy attacker.

Conclusion

Right now the scripts presented here are merely at "proof of concept" stage, although I am actively using them to monitor a few systems for myself and some of my customers. However, further work is needed in a couple of areas:

  • Better functionality for scheduling scans across a large number of hosts, including the ability to have scans run on some sort of random schedule so that they don't happen at exactly the same time every day.
  • Better error detection and possibly even a -n mode similar to the make command so that you can see what the script would do without actually performing the scan. I managed to destroy one of my test systems when working on these scripts because a mis-configured DECL file caused the check script to remove the entire file system of the remote machine I was scanning (remember there's an "ssh ... rm -rf ..." at the end of that script)!

Still, I hope you find these scripts useful or at least hope that the ideas presented here will spur your own thinking about the use of FIA tools at your site. Feedback and updates that you make will be gratefully accepted at my email address below. Happy hunting!

Notes and References

1. Tripwire home page -- http://www.tripwire.com/products/index.cfm

2. Samhain home page -- http://la-samhna.de/samhain/

3. AIDE home page -- http://www.cs.tut.fi/~rammer/aide.html

4. Perlman, Jason. 2004. "Using FCheck". Sys Admin, 13(12): 26-28. http://www.samag.com/documents/s=9426/sam0412e/

5. Hal's AIDE info -- http://www.deer-run.com/~hal/aide/

Hal Pomeranz (hal@deer-run.com) is an independent computer security consultant who has spent more time analyzing file systems than he cares to think about. No mugwumps were harmed in the creation of this article.