Eric W. Sink received a BS in Computer Science from the University of Illinois. He is currently employed as a Software Engineer with Spyglass, Inc. in Savoy, Illinois. Eric may be reached by email at esink@spyglass.com.
SENTINEL, developed by Virtual Technologies, Inc., is a tool that helps detect and locate memory usage errors in C/C++ programs. SENTINEL consists of a library which, when linked into your application, intercepts many of the standard functions and replaces them with equivalent versions that perform various kinds of error checking. The intercepted functions include memory allocation and string functions, among others. At run time, SENTINEL produces an easily readable report of the errors found in the application.
This article will present an overview of SENTINEL's functionality and examples from my experience with this tool.
Motivation for Trying Sentinel
For me, the motivating factor for ordering SENTINEL was to find a specific bug. I was just beginning alpha testing on a 140,000 line data analysis application that imported data from a number of different file formats. The application contained a module to read each file format and stuff the data into a struct called GenericDataFile. After importing a file, the application carried on its business with GenericDataFile and passed the struct pointer back to the import code so it could free the memory. Everything was working nicely except that importing one of the file formats consistently caused the application to crash. A few minutes with a source debugger convinced me that there was an error in memory management and that I was in a for a long and tedious needle-in-a-haystack search for the problem. I decided to ignore the problem for a couple of days, order SENTINEL, and give it a chance to prove itself.
Getting Started
The installation of SENTINEL presented nothing out of the ordinary. SENTINEL consists primarily of a library (libsent.a), a header file (sentinel.h), and a configuration file (sentinel.cf). By default, the following files are installed:
/usr/lib/libsent.a /usr/include/sentinel.h /etc/sentinel.cfTo start using SENTINEL, link it into an application by adding "-lsent" to the link line.After the SENTINEL library is linked into an application, the error checking facilities are ready for use.
Leak Reporting
One of the primary features of SENTINEL is its ability to produce a report of "memory leaks." Since SENTINEL replaces malloc and free, it supervises all dynamic memory allocation. After execution of the application has completed, SENTINEL locates all unfreed blocks of memory and compiles a report similar to the one shown in Figure 1.When I started examining these memory leak reports, I found it tedious and time consuming to weed through all the uninteresting leaks to find the leaks which were actually related to my application. Fortunately, SENTINEL has a number of configuration parameters to improve the signal-to-noise ratio in leak reports. The SE_EXCLUDE parameter may be used to exclude specific leaks. SE_LEAK_ONCE tells SENTINEL to group all occurrences of the same leak as a single leak. SE_LEAK_MINSIZE excludes leaks which are too small to be a problem. The manual explains these and other leak-related configuration parameters in the reference section.
Other Access Errors
SENTINEL detects and reports many other kinds of memory access errors, including reading or writing outside of allocated boundaries, and accessing memory which has already been freed. SENTINEL helped me find a couple of bugs. First, it helped me find a place where we had incorrectly zero-terminated a string. Our code was writing one byte farther than it had allocated. SENTINEL catches this kind of error by allocating a little extra memory before and after each block. The extra space is pre-filled with a special byte value, so SENTINEL can determine later if the code has trampled outside of its bounds.Second, SENTINEL detected a buggy function which was attempting to access freed memory. The function was trying to dispose of a linked list, and was equivalent to the following fragment:
node *n; n = head; while (n) { free(n); n = n->next; }SENTINEL fills newly freed blocks with a special byte pattern designed to cause a segmentation fault when dereferenced. It was easy to recognize that byte pattern in my debugger, and know that I'd accessed a pointer within freed memory.
Library Functions
In addition to intercepting standard functions, the SENTINEL library provides a number of utility functions useful in debugging an application. Many of these utility functions are designed to be called from either the application itself, or from a source debugger such as dbx. Some of these functions are used to dynamically configure SENTINEL's other features. For example:
/* Send all debug message to a certain file */ SeSetOption (SE_DEBUG_FILE, "/tmp/mydebug.out"); /* I don't plan on freeing this pointer ever */ SeNotALeak(ptr);Other library functions are functions used by SENTINEL itself, and made available to the programmer because they are handy. My favorite one is the ability to print a stack trace at any time, using the SeFwriteStark function as shown:
void *stk; stk = SeGetCurrStack(0); SeFwriteStack(stderr, "Stack:", stk);Another nice feature is discretionary debugging, provided by the SEDBG macro, defined in sentinel.h. This feature works in conjunction with a configuration parameter called SE_DEBUG_LEVEL. For example, the statement
SEDBG((30, "counter = %d", counter);has the following effect: if SE_DEBUG_LEVEL is set to at least 30, then the given message will be printed to the debugging file. By modifying the debug level from the configuration file, you can turn debugging output on and off without recompilation. SENTINEL reads the configuration file (usually etc/sentinel.cf), at run time. The file is simple to use, but contains too many parameters to mention here.
Software WatchPoints
A number of the functions provided by the SENTINEL library are associated with a feature called watchpoints. By setting a watchpoint over a certain range of memory, you can have SENTINEL alert you when that memory has been modified. This is useful when you know that a variable is being clobbered, but you're not sure where or when. The watchpoint functions are most convenient when called from a debugger:
(dbx) print SeAddWatchPoint (&nrows, 4, "write");The above example places a watchpoint 4 bytes long over a variable called nrows. SENTINEL checks all watchpoints each time one of its intercepted functions is called. For example, in the following code fragment, because the strcpy function is replaced by SENTINEL, the watchpoint write will be caught as it actually happens:
char buf[32]; strcpy(buf, "No trespassing"); SeAddWatchPoint(buf, 32, 2); strcpy(buf, "Boom"); /* SENTINEL warning occurs here */Of course, it's quite possible that your watchpoint may be modified without calling a SENTINEL intercepted function. In these cases, SENTINEL will warn you of the change the next time it checks your watchpoints (at the next call to a SENTINEL function). It is also possible to force a check at any time, by calling SeCheckWatchPoints.
Description of New 1.4 Features
Most of what I have written thus far comes from my experience with SENTINEL 1.3.5. By the time you read this, the current shipping version will be 1.4, with a number of new features which I have had the opportunity to see in beta test.SENTINEL now comes with an executable program called "sentinel" which supervises the link process to make sure all standard functions are replaced correctly. This feature allows you to simply prepend "sentinel" to your link line, and all intercepted functions will be properly replaced.
Memory leak reporting is better, now offering two basic kinds of leak reports plus a hybrid. The "match" report is the same report as was offered in 1.3, listing all unfreed memory blocks. The "garbage collection" report will locate all unfreed blocks of memory unlikely to be referenced. To produce this report, SENTINEL scans the program's entire address space, searching for pointer references. If a dynamically allocated block is targeted by a pointer, SENTINEL does not considered it a leak. The garbage collection report generally contains far fewer false positives than the match report.
The "combined" report (default) is a hybrid of "match" and "garbage collection" reports. This report lists all unfreed memory segments, and for each segment, identifies whether or not a pointer to that segment still exists. If a pointer does exist, SENTINEL reports where that pointer was found.
SENTINEL 1.4 now supports a "memory allocation" report, which lists all the places where memory is allocated, how many times, and how much.
It also includes a number of new configuration parameters, which instruct it to place timestamps in warning messages, and to send its warning messages by email.
Version 1.4 also enhances the debugging macro facility, introducing the concept of debugging classes, allowing groups of your messages to be turned on or off as a whole. The new SEDBG_LOC macro automatically includes its source line location in the message output. The SEDBG_CODE macro conditionally executes a block of C code based on the value of SE_DEBUG_LEVEL.
With the 1.4 release, VTI has introduced a license management scheme that allows them to make SENTINEL available for short term evaluations with a single license code. With each purchase of SENTINEL, an evaluation code is included in the package, allowing the user to get started immediately. Once a permanent license code has been obtained from VTI, SENTINEL is available for unlimited use on a single workstation.
Finally, version 1.4 is faster than the 1.3 releases. In general, a "sentinelized" application is slightly slower than normal, because of all the error checking. Although you would certainly not want to ship all this error checking code in your final product, the VTI license does allow you to leave SENTINEL linked in with your released binaries. This may be helpful in the beta testing period, since you can instruct beta testers to activate SENTINEL and send you any error output (or have it automatically emailed).
Origins of SENTINEL
SENTINEL originated from a freely available software package called dbmalloc, which was developed and distributed by one of the developers of SENTINEL. In fact, dbmalloc is still available from comp.sources.misc in volume 32. SENTINEL contains more features than its freeware ancestor.VTI offers a paper which discusses the differences between SENTINEL and its freeware ancestor. Copies of the paper may be requested by sending email to info@vti.com.
Product Information
SENTINEL is available on multiple platforms, including most UNIX systems. Pricing ranges from $595 to $1895. For more information, contact:Virtual Technologies, Inc.
46030 Manekin Plaza, Ste. 160
Dulles, VA 20166
703-430-9247
FAX: 703-450-4560