A New Approach to Account Activity Monitoring

Owen Stevens

My standard /etc/motd has always stated something like "Use of this system constitutes consent to monitoring..." The truth is that until recently, almost no monitoring was taking place. Sure, I use an endless variety of security tools and mechanisms to keep potential Bad Guys out of my systems, but what about the Good Guys? Once a user is authenticated, the motd is little more than an empty threat.

In my situation, there are three good reasons to monitor shell account activity. The first is, of course, security -- shell accounts are occasionally issued to consultants, vendors, and frequently to local end users. Although all of these users are essentially trusted, I cannot guarantee that their accounts will not be compromised or improperly used. Second, the ability to monitor account activity enhances my team's efficacy when troubleshooting end user sessions. Third, account monitoring satisfies some auditing requirements that are specified in corporate policies. Conversely, there is no good reason not to implement this functionality, as shell users on my production systems do not have any reasonable expectation of privacy.

During my time as a sys admin, I have often found myself trying to re-trace a user's steps. Typical Unix system configurations do not lend themselves well to this type of forensic investigation. There are only four basic tools at our disposal: the lastlog (or equivalent), shell history files, the sudo log (if applicable), and the find command.

I usually start by examining the lastlog to determine the relevant login times of the account in question. Although reliable, the information provided is minimal. I then refer to the shell history file associated with the account. This method has several major drawbacks. History files do not timestamp entries or separate sessions, therefore it can be difficult to correlate shell history with the information contained in the lastlog. Users obviously need write permissions to their own history files, a necessity that all but eliminates their usefulness as a monitoring component. The sudo log is only useful when the account has access to privileged commands, although it does a decent job of logging both successful and failed attempts in those instances. Both the sudo log and the shell history file are similarly limited when they record the execution of an interactive command. When interactive commands are executed, only the command itself is logged; any subsequent interaction is not recorded. In the cases of missing or incomplete shell history files, I can use the find command to list files owned or writable by the account, that have been modified since a given time. At this point, I have completely given up any pretense of monitoring and am merely searching for clues.

To live up to the claims I implied in /etc/motd, I need to dramatically improve my ability to monitor user account activity. As I set out to accomplish this, I soon became aware of a significant limitation -- any solution will inevitably involve some process invoked by the user at login. It follows that the user will thereby have write access to any files created by such a process, as well as the ability to terminate the process. Resolution of this core issue precedes any reasonable claim of progress. I considered various options, but each one invariably led me back to the same paradoxical problem, until I eventually devised an acceptable response.

The Access Server

I decided to "wrap" access to the server in question by introducing an intermediary server to which users must connect as the prerequisite path to their intended destination server. This "access server" can be a small PC running Linux, on which the user has almost no privileges. The access server performs three essential duties: it handles user authentication, session monitoring, and connectivity to the "destination server". All of these should occur with complete transparency from a user perspective. I began by setting up user accounts on the access server to match those on the destination server. Because I don't want the users to have to enter their password twice, I set up ssh keypairs that use a null passphrase for each account. Once a user authenticates on the access server, a connection to the destination server can be established without any further interaction from the user. This procedure is often referred to as "passwordless ssh", which I won't cover in this article because suitable tutorials are readily available on the Internet.

Session monitoring is accomplished on the access server with screen, which is invoked by the user at login. To ensure that the user has no other privileges on the access server, I need to make some rather debilitating changes to /etc/bashrc and /etc/profile. I'm okay with doing this because the access server has no other role in my infrastructure. In addition to these changes, I also need to disable the sftp subsystem in the sshd_config file, and ensure that the access server will not run any network services in addition to sshd. I have now restricted all non-root users to a screen session, which logs their activity and initiates a connection to the destination server. When this process exits, the user is logged off the system, and because exec is used, there is no opportunity for the user to break into a local shell. Although users have write permissions to their own logs, there is no available interface to these files. The relevant changes in each file are shown below. /etc/bashrc:

# /etc/bashrc 
[...] 
# Effectively disable scp by echoing something to STDOUT at login: 
/bin/echo "scp disabled"
            

# /etc/profile 
[...] 
if [ "$USER" != "root" ]; then 
        exec /usr/bin/screen -l -L -q -s \
          '/usr/local/bin/connect.sh' -c /etc/screenrc 
        exit 
fi 
# 

# /etc/screenrc 
[...] 
logtstamp on 
logfile "/var/log/screenlogs/screenlog.%c:%s"
#

#!/bin/bash 
# /usr/local/bin/connect.sh 
/usr/bin/ssh $USER@destination.domain.net 
# 

The Destination Server

On the destination server, there are a couple of slight changes to make. I'd like to set the maximum number of concurrent sessions to one per non-root account, to ensure linear creation of the screen logs on the access server. I'll also use a host-based firewall on this server to deny ssh connections that do not originate from the access server. This step guarantees that the access server is used every time. Due to excellent documentation of host-based firewalls available on the Internet, I'll skip discussing this step any further. Here is the stanza in /etc/profile that limits the number of sessions:

# /etc/profile 
[...] 
# Only one concurrent session per non-root account allowed 

readonly INSTANCES=`w|awk '{print $1}'|grep -w `whoami`|wc -l' 
if [ "$INSTANCES" -gt 1 ] && [ b$USERb != brootb ]; then 
  stty susp ^- intr ^- eof ^- quit ^- 
  echo "You are already logged in."
  sleep 5 
  exit 
fi 
# 
    

Typically, vendor accounts may have sudo access to specified commands, and they may have write access to application configuration files, and sometimes database access. For each account, I identify any pertinent files and directories. The level of granularity varies considerably between accounts; of course, each particular situation must be analyzed. I'll demonstrate a basic example configuration of /etc/aide.conf. If the vendor1 user modifies any file in his home directory, or modifies his crontab, then this activity will be detected:

# Configuration file for AIDE 
@@define DBDIR /var/lib/aide 
database=file:@@{DBDIR}/aide.db.gz 
database_out=file:@@{DBDIR}/aide.db.gz 
gzip_dbout=yes 
verbose=20 
report_url=file:/var/log/aide.log 
report_url=stdout 
[...] 
# Custom Rules 
FILE = p+i+u+g+n+s+b+m+c+md5 
DIR = p+i+u+g 

# Files and directories to monitor: 

=/home/vendor1$                         DIR 

/var/spool/cron/vendor1                 FILE 
# 

File: /var/spool/cron/vendor1 
  Size     : 684                               , 629 
  Mtime    : 2006-04-21 10:31:55               , 2006-07-05 12:07:04 
  Ctime    : 2006-02-09 22:32:22               , 2006-07-05 12:07:04 
  Inode    : 7209157                           , 7209134 
  MD5      : kAe8EKx2mXQmmrSbAitp7A==          , 
N/Sr/vD3/7oQ0UnyK4fbmA= 
# 
    

Finally, I recommend configuring ntpd on both servers so that the clocks on both these systems are synchronized. This step will greatly assist any future forensic use of the screen logs. With this accomplished, I have deployed an effective combination of ssh and screen within an enforced shell environment, which records session activity in a tamper-resistant fashion. The resulting logs provide detail superior to anything previously available, and this functionality has been deployed without interruption of service or negative user experience. My team also now has a valuable session troubleshooting resource.

I'm sure that nobody actually reads /etc/motd, but I like to think it can have a little more weight than expected. On my systems, you can take it to the bank.

Owen Stevens is currently employed as the Senior Systems Administrator for a medium-sized online retailer. He has more than eight years of experience as a Unix/Linux Administrator and runs a small consultancy. Owen can be contacted at: ostevens@gmail.com.