C/C++ Users Journal March 2004

Editor's Forum

Like many a professional programmer, it took me a while to appreciate the need for proper testing. I managed to turn out more than one successful software product before I got religion, but in hindsight I shudder at the risks we took every time we shipped a release. Even more embarrassing, I taught a number of seminars on the art, craft, and discipline of testing, long before I had any real notion of how to do it right in the real world. Of course, in those days I still thought that the ability to wing it, and snow your customers and/or students, was one of the higher virtues.

The business of testing software suffers from a number of intrinsic problems:

• It is boring. Many of us are drawn to programming, I believe, out of a love of problem solving. It's neat to make a computer look smart with an economy of bytes and microseconds. It's tedious to write all the code needed to show up any stupidities that might have crept in along the way.
• The people who know the code best have the hardest time writing tough tests. Back when I was writing compilers, my first acceptance test was to get the compiler to compile itself. I naturally avoided using any constructs in the compiler that it was likely to get wrong. And even when I set out deliberately to write stress tests, somewhat later in my career, the temptation was ever present to go easy on the code I knew to be the most fragile.
• The people who are willing to write code often lack the drive and/or skills to do it right. I've always been passionate about programming, and I naturally gravitate toward others who feel the same way, but our trade does attract people who just like the pay. Managers stick them on testing projects and they don't complain. They also don't test very hard.
• The people who like to write code are often perverse. Yes, it can be useful to have someone who delights in testing everything to destruction. But you can't let that mindset color an entire project, or you end up spending too much time refining corner cases that customers just don't care about.

Testing is most definitely an art form. It is well understood that any nontrivial piece of code can never be tested exhaustively. It is less well understood how to select just those anecdotal tests that are most likely to show up problems. Here, more is definitely not merrier. If a single flaw causes 700 test failures, a test suite is less useful than one that reports just one or two failures. And if a suite has to run all night it is less useful than one that does its job in minutes.

Testing is also most definitely a craft, if not a full-fledged engineering discipline. My company has spent most of the past year developing and refining test methods for math functions. We're nowhere near as clever as the past masters, such as Cody and Waite, but we've learned a lot from them. And we've learned to develop mechanical methods for generating tests, so we don't have to be that clever either. (We've found an embarrassing number of lapses in our own code. But we're now much better armed for competitive analysis, too.)

Running tests, and dealing with test failures, has become an engineering discipline. Like most disciplines, you see the need for it most clearly when it is not present. Beware the shop that doesn't run tests regularly, particularly one that doesn't require exhaustive testing before each version freeze. Beware test suites that require much human intervention in the running, or in the interpretation, of results. Most of all, beware the manager who will excuse "just one or two test failures," rather than hunt down the root causes.

There's some good news among all these bleak observations. I'm finding more and more programmers who are good at both development and testing. Pete Becker impressed me, for example, long before we hired him, because he asked to move into testing at Borland. He reflexively develops code and tests at the same time. I'm also meeting more and more people who proudly pursue testing as a profession. For another example, we contract out much of our test harness development to John Benito. He likes to develop orderly test methods, and he likes to exercise test suites—and he's very good at both.

There's still plenty of naiveté to contend with. Just read the newsgroups for a few days and you'll see what I mean. (The posters may not be a representative sample, but they're certainly a loud and influential one.) What they don't understand about correctness testing is exceeded only by what they don't understand about performance testing. But that's a whole 'nother sermon.

P.J. Plauger
Senior Contributing Editor
pjp@plauger.com