Choosing Test Data


As implied by the title of Glenford Myers' classic book on the subject, The Art of Software Testing, choosing good test data is a combination of science and art. Most methods fall into one of two categories: "black box" or "white box."

Black-box methods of finding test data ignore the code and work from its specification, that is, from a knowledge of the task the software must perform. Myers suggests that testers should try to write one test for each equivalence class of inputs — that is, one for each range of inputs for which the program should perform the same way. For example, if a program is specified to accept up to five transactions in each batch, then it makes little sense to run three separate tests for batches of two, three, and four transactions; one test with just one of these values is probably sufficient.

Another good black-box rule is to check just inside and outside boundary values, where off-by-one errors may be revealed. Thus in this same example, a good test set would probably include tests with an empty batch, with a one-transaction batch, with a five-transaction batch, and with a six-transaction batch. It is important to include tests with incorrect data, such as the empty batch and the six-transaction batch in this example, since a well-engineered program should behave predictably even when its input is wrong.

By contrast, white-box testing involves using the code itself to help find test data. The idea is to make sure that all the code is "covered" in some way, meaning that it all has been tested. Of the several possible criteria for coverage, the simplest is "statement coverage," the goal of which is to ensure that the test set causes every executable statement in the program to be executed at least once. Stronger criteria involve making sure that each side of each branch is taken or guaranteeing that each subcondition within every compound-branch condition is evaluated to both true and false.

In doing white-box testing, the tester normally uses a test coverage tool. The tester runs a series of tests, then uses the tool to identify statements, branches, or conditions that have not yet been exercised. The tester then identifies further tests to try to get coverage as close as possible to 100 percent.

Both black-box and white-box methods have their advantages. White-box testing is easier to automate and evaluate, but may not find certain kinds of bugs, such as the omission of some required program function. Black-box testing needs more creativity, but may detect more subtle errors. Thorough testing, in OOP or otherwise, should probably involve a combination of both approaches.