Process Accounting in Solaris^TM 10

In the Unix world, process accounting refers to the capability of the operating system to track system activities by recording statistics of many kinds (e.g., what commands have been run, by whom, for how long, and how many resources they used). Process accounting was conceived (obviously enough) for accounting purposes, so that users could be billed for the resources they used. These days those tracking capabilities are more frequently used for security reasons, so that a systems administrator can easily check who did what.

In this article, I'll show how to set up accounting on Solaris 10. My sources of information are mainly manuals and books, but I've also drawn upon my own experience where those resources left some gaps. I'll also describe some improvements that may be useful in helping you perfect your own setup. I will not cover the format of the reports (because that is already covered in the documents in the references), nor will I talk about how accounting works on other Unix systems.

Some considerations apply, namely:

• Despite the fact that process accounting has been available for a long time, chances are that it won't work out-of-the-box even if you followed some kind of "official documentation".
• Accounting doesn't come for free in terms of resources; a rough estimate says that the system load could increase up to 20%.
• Log files, commands, utilities may differ from system to system; I will focus on Solaris 10. Please check the articles in the references if you wish to adapt this information to other operating systems.

For process accounting to work, accounting capabilities must be present into the kernel, and Solaris 10's kernel has them built-in. This is not enough: accounting must be explicitly activated. Solaris 10 has the /etc/init.d/acct file ready but no correspondent link in the /etc/rc?.d directories, so you will have it activated at boot only if you really want to.

Once accounting has started, you'll need to create reports on a regular basis, running the commands runacct and monacct. The runacct command is actually a script that, run once a day from the crontab, forks a lot of other small scripts, each one playing a particular part in the process of creating the reports. The monacct command works pretty much like runacct and aggregates daily reports data in monthly reports; just like runacct, it runs from a crontab entry once a month. Please note that both scripts should be run by the user adm and not by root; if you accidentally run them as root, you will mess up ownerships and permissions of the files affected by any commands run.

The reports produced by those scripts are left on the file system. It is up to you to get and read them. You should also take care not to leave them alone for so long that they fill the file system or use up all the available inodes.

It's not easy to draw the complex relationships between runacct, monacct, and the other scripts. You'll find a simplified diagram in Ūleen Frisch's Essential System Administration [1] (and yes, I do mean "simplified"...).

How to Set Up Accounting: What the Manuals Say

The following is a mix of the information you can find in Frisch and in Sun Microsystems' System Administration Guide [2]; both of these lack something here and there. Even by merging them you don't get all the information you need; nevertheless, they are a very good sources of information.

You should first make sure that process accounting starts at the system boot. To do this, you just create a couple of links to /etc/init.d/acct. The manuals say to create a hard link, but I prefer using symbolic links:

ln -s /etc/init.d/acct /etc/rc0.d/K22acct
ln -s /etc/init.d/acct /etc/rc2.d/S22acct

#####################################################################
# Information on the following entries comes from
# "Essential System Administration", 3rd Edition, by Ūleen Frisch
# edited by O'Reilly and Associates
#####################################################################
# control accounting file size
30 3 * * *      /usr/lib/acct/ckpacct
# process accounting raw data
30 4 * * *      /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log
# generate monthly reports
30 5 1 * *      /usr/lib/acct/monacct

# One sample per day
30 22 * * * /usr/lib/acct/dodisk

The next step is to update the /etc/acct/holidays file. The format of the file is slightly unusual:

• Lines prefixed with an asterisk are comments (this is nothing new in Solaris...).
• The first non-comment line is a three-field record that contains the current year, and the time of the day when prime time starts and ends.
• Subsequent non-comment lines are the holidays for the current year; each one is a two-field record that contains a date in the current year and a readable label for the holiday. Please note that the accounting system only considers the first field; the second one is only for your convenience.

* @(#)holidays  January 1, 2005
*
* Prime/Nonprime Table for UNIX Accounting System
*
* Curr  Prime   Non-Prime
* Year  Start   Start
*
  2006  0900    1730
*
* only the first column (month/day) is significiant.
*
* month/day     Company
*               Holiday
*
1/1             New Years Day
12/25           Christmas

Finally, you need to create a small group of directories; if you are using a bash shell you can make them quickly this way:

mkdir -m 750 -p /var/adm/acct/{fiscal,nite,sum}
chown adm:adm /var/adm/acct/{fiscal,nite,sum}

From the documentation, it seems that you are all set, so you just need to start process accounting by running:

/etc/init.d/acct start

How to Set Up Accounting: What the Manuals Don't Say

What's missing in the above setup is one very important point. We said that runacct and monacct run as user adm. The home directory for adm is /var/adm and... this user has no rights on it! It may not even create files or directories!

There are many ways to fix this, and I chose the following one:

chmod g+w /var/adm
chgrp adm /var/adm

The third thing is that some reports might be missing. In my case, I wanted a log of all the commands run by users along with the indication of who ran them. I tried to achieve this (and more) functionality by applying information about make that I'd learned earlier. I had just finished reading Tom Limoncelli's excellent book Time Management for System Administrators [3], which had a small section dedicated to an introduction to GNU make, so I decided to apply some things the book had taught me.

Towards the Perfect Setup: Refining your Configuration

Please note that the Solaris make command won't work, so you'll need to install the GNU version of make.

Place the file shown in Listing 1 into /var/adm and name it "Makefile". You should also change adm's crontab accordingly:

#ident  "@(#)adm        1.5     92/07/14 SMI"   /* SVr4.0 1.2   */
#
# The adm crontab file should contain startup of performance collection
# if the profiling and performance feature has been installed.
#
#####################################################################
# Information to build this file comes from
# "Essential System Administration", 3rd Edition, by Ūleen Frisch
# edited by O'Reilly and Associates
#####################################################################
# control accounting file size
30 3 * * *      /usr/lib/acct/ckpacct
# process accounting raw data
30 4 * * *      /usr/local/bin/make runacct > /dev/null
# generate monthly reports
30 5 1 * *      /usr/local/bin/make monacct > /dev/null

Understanding the makefile

In the first three lines, set three static variables that will come handy later. The subsequent three lines dynamically set three other variables:

DAILY = $(shell ls -1tr $(SUMDIR)/rprt* | tail -1)
MONTHLY = $(shell ls -1tr $(FISCDIR)/fiscrpt* | tail -1)
TODAY = $(shell date '+%d')

The last assignment to the variable MAKE is just a shortcut. I use it so that I don't have to rely on the correct setting of the environment variable PATH to use the correct version of make.

Figure 1 shows the relationships between the targets in the makefile -- the arrows show how one target invokes another. We'll go through them not in the order they appear in the makefile, but from the bottom of the figure to the top (from the more specific ones to the general ones).

• The target report.daily does some sanity checks and, after that, sends the latest daily report to root. Of course, you can send your report to anyone you like -- just customize the makefile.
• Similarly, report.monthly sends the latest monthly report to root.
• runacct.log sends to root the contents of the fd2log file if it's not zero in size.
• reports calls unconditionally the target report.daily and also calls the report.monthly target if it's the first day of the month.
• monacct calls the monacct command and, once it's over, calls the report.monthly target to send the report that monacct just produced.

The runacct Target

In the runacct target, we are not just running the homonymous command. The target starts by running the acctcom command and sending the results to root. The acctcom command, when run without any options, sends a report of the commands run in the past 24 hours, saying which commands were run, by whom, whether they were run with superuser privileges, when they were run, for how long, and also provides a summary of the resources used by the process. In my opinion, this is an important report, but the standard system doesn't create it by default. For more information about acctcom see the online manual.

After that, you will find the classical execution of runacct, followed by the call to the two targets runacct.log and report.daily. By invoking these targets, you will get the daily report and the log of runacct errors.

Further Enhancements

You could optimize this even more, for example, by collapsing the three calls in adm's crontab in a single call to make and letting make look at the date and decide whether it needs to run runacct or both runacct and monacct. These enhancements are left as an exercise to the reader (hint: just look at the reports target and extend it, that's pretty easy).

You may also want to add a cleanup job that could be run every day or on the first day of every month to remove the older reports from the file system (or to create a compressed archive of them). This can also be achieved by customizing the makefile, for example, by adding a new target to it and calling it automatically when appropriate.

The number of things you can do depends just on your imagination and needs; feel free to use this makefile as a framework to build upon.

Conclusion

As you can see, process accounting can help you monitor the activities on your system and produce many different reports. These reports are useful for statistics purposes or to investigate incidents. Accounting is built in the system and is also a robust and complete solution, because it gathers information from many sources down to the kernel level. On the other hand, it does add load on your system, and you need to be aware of that if you are going to enable accounting on a busy machine. Accounting is also not fully documented, but I hope to have addressed that problem with this article.

Acknowledgements

The author thanks Tony Mobily from Free Software Magazine (http://www.freesoftwaremagazine.com/) for proofreading this article and catching a number of mistakes.

References

1. Frisch, A. 2002. Essential System Administration, Third Edition. O'Reilly & Associates.

2. SUN Microsystems: System Administration Guide Advanced Administration. In the Solaris 10 System Administrator Collection -- http://docs.sun.com/app/docs/doc/817-0403

3. Limoncelli, T.A. 2005. Time Management for System Administrators. O'Reilly & Associates.

4. Siever, E., S. Spainhour, J.P. Hekman, and S. Figgins. 2000. Linux in a Nutshell, Third Edition. O'Reilly & Associates. (Note that some information about make and makefiles came from this book, too.)

Marco Marongiu graduated in applied mathematics in 1997; he's now a full-time system administrator for a European ISP. He's also a Perl programmer, technical writer, and lecturer by passion and is interested in Web- and XML-related technologies. Marongiu has been a Debian User since version 1.1.10 and helped found the GULCh Linux Users Group (Gruppo Utenti Linux Cagliari), the first one in Sardinia. He recently became a father to his first son, Andrea; and he's been trying to reorganize his life ever since, so that he can start writing technical articles and holding seminars again.