Norman Wilde has been writing and testing programs since the Fortran II days of the early sixties. Since 1987 he has taught in the Masters of Software Engineering program at the University of West Florida and has done research on software maintenance and on the impact of object orientation on maintenance. With coresearchers from Bellcore, he published an article "Maintaining Object Oriented Software" in the January issue of IEEE Software. Norman can be reached on Internet as wilde@cs.uwf.edu.
Testing units individually before integrating them into a software system is critical to efficient system development. In traditional systems, the units are subroutines (for example, C functions), which can be first tested alone, then integrated into larger and larger collections until the system is complete. Detecting and correcting errors is much easier at the unit level than at later stages of integration because the amount of code to be examined is much smaller.
For object-oriented programs, the important units are not subroutines but classes of objects (the C++ member functions that act as subroutines are typically too short to warrant individual testing). Unit testing, in this case, means exercising thoroughly the life-cycle behavior of an object: its creation, its state changes, and its destruction.
Unit testing entails writing test driver programs — i.e., small programs that set up the data needed to test the unit, call it, and then print out the results of each test. Building test drivers for objects can get complicated, because the object to be tested will often have objects of other classes as members or as member function parameters.
Consequently, a good object test driver must construct quite a variety of objects. The example in Listing 1 shows a header for several simplified classes that might appear in an invoicing application. An invoice contains a client and a list of items. The total discount for the invoice depends on the type of client and on the product codes of the items. To test the discount calculation, you need to construct Client and Item objects that are used to create the Invoice object and to exercise the Invoice member functions.
A good object test driver must also step through a variety of inputs, trying a range of values for each object (errors often appear only at extreme values, as for example, when the list of items is empty or when it contains just one item — for a discussion of appropriate test data, see the box entitled, "Choosing Test Data"). Finding subtle errors requires checking combinations of circumstances, such as a special product code that appears in a list with just one item. Enumerating the different combinations that should be checked and writing the driver code to generate them can be very tedious.
Using a Little Language for Testing
To simplify the process of building object test drivers, you can define a "little language" that lets you specify and generate test drivers for the objects in your program.Figure 1 shows the components of the testing system. The driver generator language syntax file DGEN.SYN defines the little language. A parser generator (AnaGram in this case) converts the syntax file into a C program called DGEN.C, which compiles into the driver generator DGEN.EXE. The driver generator reads test specifications written in the little language and produces a C++ test driver, which is compiled and linked with the code for the objects being tested.
Listing 2 shows a sample test specification for tests of the Invoice object class from Listing 1. The little language allows C++ style comments, as shown in the first two lines. The block within braces that follows will be simply copied over into the test driver program so that the program will include the headers it needs.
There are really only two important kinds of statements in the little language: set variable declarations and runtest statements. A set variable declaration names a set of objects that the test driver will choose from in performing the tests. For example, the declaration
Client someClient { [Client(RETAIL)] [Client(WHOLESALE)] [Client(FOREIGN)] }(from Listing 2) describes a set named someClient with three Client objects of different client types, and gives initializers for each object. The declarations that follow this in Listing 2 describe a set of integers called numItems and two identical sets of Item objects called someItem and otherItem. The final set, called theInvoice, has just one member. Note that its initializer uses the someClient set. During each actual test run, the test driver will choose just one member from the someClient set to construct the invoice object. Thus the test driver will do the work of creating objects which contain other objects in different combinations.Each runtest statement specifies a series of actual test runs. For example, the statement
runtest "r1" combining someClient theInvoice numItems someItem { ... }tells the driver to select all possible combinations of values from the set variables someClient, theInvoice, numItems and someItem. For each such combination, the code in the block will be executed to test the totalDiscount calculation. As Listing 2 shows, this code writes out information about the objects being used and sends a series of addItem messages to the invoice, followed by a final totalDiscount message.Once you have defined the little language, you can generate flexible test driver programs quite easily. One important feature of this method is that the test specification file provides simple and compact documentation of each test. When the system changes in the future and tests need to be rerun, the test driver can be quickly recreated. It is good practice to make sure that key output from each run is also saved for comparison with future test results.
The Little Language for Driver Generation
Listing 3 shows the most important part of the AnaGram grammar of the driver generator. (The complete syntax, alone with other files for this technique, is available on the C Users Journal's monthly code disk and via Internet and CompuServe). A test specification simply consists of a series of statements followed by end of file. A statement is either a block of embedded C++ code, a runtest statement, or a declaration. There is one kind of declaration for each class of objects to be used in the test; the example includes Item, Client, and Invoice objects as well as integers.The driver generator parses the test file and identifies each kind of statement. When it encounters a declaration, it calls the addTVars function to add the declared set variables to a symbol table. Strings stored with each variable give its type and its list of initializers. When a runtest statement is parsed, its list of variables is passed up to the makeComb function along with its C++ block. makeComb emits C++ code to construct every combination of the variable initializers and send them to the C++ block.
The Generated Test Driver
Figure 2 shows a structure chart for the test driver generated from Listing 2. A hierarchy of functions is generated for each runtest statement in the test specification, with each function responsible for setting the value of one of the set variables in the statement. Each function contains a local array of objects for its variable, declared using the initializers in the corresponding variable declaration.The main function generates an integer vector, vals, that represents a combination to be tested. Each other function in the chain uses the value of one element of vals to select a single object out of its local array and pass it to the function below. Thus each function has available a single value for each of the variables above it and can use those values to initialize its own objects. The bottom function contains the block of C++ code from the runtest statement that actually does the test. It has available values for all of the variables. Thus the main function simply needs to generate all possible vectors vals and call the top function in the chain for each one.
There are a couple of variations to be considered in using this technique. First, the objects can be passed down the chain either by value or by reference. The C++ default is to pass function arguments by value, which means that a copy of each argument will be made using a copy constructor; if you don't code a copy constructor, then the C++ compiler creates one for you, and it may sometimes have side effects that surprise you!
You control the selection of pass-by-reference or pass-by-value in the driver generator syntax file. The addTVars function takes two string arguments that are used in declaring the types of objects. The first is used in declaring the array of objects; the second, in declaring the arguments passed between functions. Thus, for example, in Listing 3 the line
addTVars (vl, "Client","Client&", il)says that for Client set variables the array will be declared using a type of "Client" while the function arguments will be declared as "reference to Client". The decision can thus be taken differently for each class of objects. If you pass arguments by reference, then you will have more control over object creation and destruction during tests. However, I have uncovered several memory management bugs in my objects using pass-by-value, since the object constructors and destructors were thoroughly exercised.Another question to consider is whether you really want to try all combinations of values from the sets you have declared. The number of test cases can be very large, and although the running of the test cases is automatic, you still must do the hard work of carefully examining the output for errors. Sometimes it may be sufficient to vary the different objects one at a time, holding the variables not being varied at some standard value. The complete driver generator I have been using allows this option.
Conclusions
You can't completely eliminate the hard work required for unit-testing your objects: you must think carefully about the values you want to use and the sequences of messages you will send to each object; most important, you must review the output of each test very carefully to ensure that there is no concealed error. By using a little language and a parser generator, however, you can automate a considerable part of the work of setting up and running tests. You still have a lot of work, but you'll be able to devote more effort to inventing good test data and less to the mechanics of the testing process.
Further Reading
The classic book on software testing is The Art of Software Testing, by Glenford Myers (New York: John Wiley and Sons, 1979). While somewhat old, it is probably still the best source of practical hints on testing software.Although several sources describe the useful technique of developing "little languages" to solve problems, the method still is not used as often as it might be. Jon Bentley gives several examples in Chapter 9 of More Programming Pearls: Confessions of a Coder (Reading, MA: Addison-Wesley, 1988).
Parser generators usually are discussed at a level of theory that makes it hard to see how useful these tools can be for tasks other than just compiler writing. John Levine, Tony Mason, and Doug Brown describe the yacc-based tools in lex & yacc, (Sebastopol, CA: O'Reilly & Associates, 1992). Another readable source is chapter 4, "Introduction to Syntax Directed Parsing," of the AnaGram User's Guide (Jerome T. Holland, 22 Forty Acres Dr., Wayland, MA 01778).
Sidebar: "Parser Generators"