Article apr2006.tar

Archiving Korn Shell History Files

John Spurgeon and Ed Schaefer

Shell history gives users the ability to manipulate commands previously entered at the command line. It is not an auditing tool. Nevertheless, systems administrators often resort to looking at shell history files to trace user activity. When used as an audit utility, shell history has serious drawbacks. For example:

  • Once a shell history file contains the maximum number of commands, old commands are removed as new ones are entered.
  • History files may be modified by a user who is trying to cover his tracks.
  • Timestamps are not available to determine when a command was executed.
  • History files are typically stored in the user's home directories making it difficult to process the information.
  • It can be tricky to tell who executed certain commands if someone used su to become another user, such as root.

In this column, we present a collection of shell scripts that attempts to mitigate some of these drawbacks. You shouldn't rely on our solution to catch users bent on mischief. Tools like Solaris's Basic Security Module (BSM) are better suited for that type of monitoring. However, you might find that our scripts significantly increase the value of your shell history files. Also, the resulting information might be easier to work with than the massive amounts of data that can be generated by a system like BSM.

Program Design

The Korn shell provides the ability to retrieve a user's command history, which is stored in a file identified by the HISTFILE environment variable. The number of commands stored in this file is defined by the value of the variable HISTSIZE. Other shells may provide similar shell history capabilities, but our focus is limited to the Korn shell.

When a user logs in, we set her HISTFILE and HISTSIZE variables and make them read-only. When the user logs out, the contents of her history file is copied to a file in an archive directory with limited access. We also set up a cron job that periodically inserts timestamps into open shell history files so the time a command was executed can be estimated.

Although the design is simple, the actual implementation is somewhat complex. The rest of the column describes the environment and the script listings.

The Environment

Our last column, "Implementing Standard Login Scripts", discussed customizing login scripts for interactive shells. You may want to review this discussion if you are not familiar with the way the Korn shell handles login scripts. On our systems, ENVFILE is set to /etc/.kshrc. This script (Listing 1):

  • Calls /etc/setenv_user (Listing 2), which sets several variables including REAL_USER and TERM_USER, which are relevant to the way we manage history files.
  • Calls /opt/logins/bin/sh_history.initialize script (Listing 3), which sets HISTFILE and HISTSIZE.
  • Sets the logout script to /etc/.logout (Listing 4).

A repository exists for both open and archived history files. The location of the repository is defined by a read-only environmental variable, HISTORY_DIR, which is set in all the scripts by sourcing the sh_history.globals file (Listing 5). We've chosen the directory /var/sandbox/.sh_history. The following directories under $HISTORY_DIR must also exist with the following permissions: 

drwxr-x---   2 root     people     17920 Dec 20 14:56 archive
drwxr-x---   2 root     people      2048 Dec 20 15:36 open
drwx------   2 root     people      1024 Dec 20 15:36 pids
sudo Setup

On our systems, every user with command-line access belongs to a group called people. We use this group in our sudoers file to allow users to execute certain commands as root. Within the sudoers file, we set the following command alias and user specification: 

Cmnd_Alias PEOPLE_COMMANDS = \
/opt/logins/bin/sh_history.touch, \
/opt/logins/bin/sh_history.clock_in, \
/opt/logins/bin/sh_history.wipe, \
/opt/logins/bin/sh_history.list

%people ALL = NOPASSWD: PEOPLE_COMMANDS
Explaining sudo is beyond the scope of this article. For more information about sudo, see: 

http://www.gratisoft.us/sudo/sudo.html
Sourcing the sh_history.initialize Script

When a user logs in, the program /etc/.kshrc calls sh_history.initialize. Sourcing this script sets the values of HISTFILE and HISTSIZE. The HISTFILE variable is a file name composed of strings representing the date, process id, and user ids. The full pathname of a typical history file looks like this: 

/var/sandbox/.sh_history/open/051221-095155-24533.eds-as-eds
Someone who uses su to become root might generate a history file like this: 

/var/sandbox/.sh_history/open/051221-095155-24533.johns-as-root
Once the history file name is established, a call to the sh_history.touch script (Listing 6) ensures that the history file exists and sets the owner, group, and file permissions. Then the date, user ids, and the output of the command /usr/bin/who -m | cut -f2 is written to the history file.

Finally, sh_history.initialize calls the sh_history.clock_in script (Listing 7), which "clocks in" the history file. History files that are clocked in are in use by their respective user id. Files not in use can be archived.

Timestamping History Files

Periodically, a root cron job executes the /opt/logins/bin/sh_history.timestamp script (Listing 8), which appends a string containing the current date and time to the end of each open history file. For example, the following entry in root's crontab file calls sh_history.timestamp every 30 minutes: 

0,30 * * * * /opt/cron/root/sh_history.timestamp
The timestamp script obtains a list of active shell history files by executing the /opt/logins/bin/sh_history.list script (Listing 9). The sh_history.list script returns the list by simply executing a find on the $HISTORY_DIR/open directory.

If any files are returned, this string: 

"# Open: $(date)\n\0000\c"
is appended to each file. On our systems, the Korn shell includes some special formatting characters when appending lines to a history file. Figuring out what the shell was doing required a bit of ASCII character sleuthing. We eventually determined how to mimic the behavior of the shell by adding the string "\n0000\c" to the end of a line. This improves the readability of the history file when it is being used for its intended purpose.

Archiving the Shell History Files

The /opt/logins/bin/sh_history.wipe script (Listing 10) archives history files that are no longer in use. It executes when called by the sh_history.timestamp cron job. The script is also called by /etc/.logout, which executes each time a user logs out.

The sh_history.wipe script generates a list of process ids using the names of the files in the $HISTORY_DIR/pid directory. The ps -p command determines whether the process is active; if not, the corresponding file (a hard link) in the pid directory is deleted. Deleting the file reduces the number of links -- presumably to one -- which triggers the sh_history.wipe script to archive the history file.

Note that when a user is logging out, the sh_history.wipe script sees that that the pid for that user's shell is still active, so the user's shell history file is not archived until the next time sh_history.wipe executes.

The script next finds all the files in the $HISTORY_DIR/open directory. Any file with only one link is ready to be archived.

When a history file is archived, the contents of the file are copied to a new file in the directory $HISTORY_DIR/archive. The unreadable formatting characters are removed, and another string is appended to the file indicating when the file was archived. Also, the proper ownership, group, and permissions are set on the archive file.

Although the lines are commented out, if a user has logged in as root, we generate an alert. For more information about this technique, see "Managing Enterprise Alerts", Sys Admin, April 2004.

What's in the Tarball?

The scripts explained in this article are located in the /etc and opt/logins/bin directories. The tarball is created with relative paths to facilitate placement in your environment. If you choose not to use /opt/logins/bin directory as your script repository, you will need to update the scripts accordingly.

References

BSM Analyzer, Sun Help Desk, John Richardson, 10/20/2003 -- http://www.sunhelpdesk.com/jmr_BSM_Analyzer.htm

John Spurgeon is a software developer and systems administrator for Intel's Factory Information Control Systems, IFICS, in Aloha, Oregon. Outside of work, he enjoys turfgrass management, triathlons, ultra-marathon cycling, and spending time with his family.

Ed Schaefer is a frequent contributor to Sys Admin. He is a software developer and DBA for Intel's Factory Information Control Systems, IFICS, in Aloha, Oregon. Ed also hosts the monthly Shell Corner column on UnixReview.com. He can be reached at: shellcorner@comcast.net.