C/C++ Users Journal October, 2004
"When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure it and when you cannot express it in numbers, your knowledge is a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science."
—William Thompson, Lord Kelvin
Probably you've run across some variant of this quote before. I repeat it in its entirety for the simple reason that you've probably run across some variant of it, and not the real thing. I've seen it subtly distorted to support arguments against measuring anything related to education. I've seen it truncated by nonscientists to avoid the obvious and perhaps odious comparisons. I've also seen it blatantly ignored at times by people who should know better, including me.
What passes for measurement in the business of software development is the thing we call "testing." You can't know that a program works at all until you actually run it and poke at it a bit. You can't know that it does what's intended unless you have some notion of what outputs to expect for a set of inputs—and then you have to actually generate test cases for those inputs, run the cases, and look at the output. And you can't know that a program is reasonably robust for a large assortment of inputs, including malicious corner cases, unless you test those, too.
It is the sheer complexity of software that makes all these things true. We can't possibly predict exactly what a compiler will do with our code, or what an operating system will do with the executable, or what the rest of the environment will also bring to the party. Each is a subsystem way too complicated for us to model completely in our heads. The odds that we'll get everything right the first time, or even after a trivial change, are way too small to chance.
And yet too often we do. Sometimes a kind of arrogance creeps in. Too often, I've seen people freeze a spec without having written a line of code, much less run any tests, to see if it holds water. Sometimes we're in a hurry. Too often, I've seen people perform just a few spot checks and then write off the testing effort as complete. And sometimes we just get lazy. Too often, I've seen people who may be good at design and coding, but who lack the special skills needed to write and debug a comprehensive set of tests, so they don't even try very hard.
The excuses are legion. We sneer at the artificiality of most software metrics (and rightly so sometimes, just like those educators who try to protect students from the tyranny of test scores). We ship buggy software in the name of rapid prototyping, or—these days—agile software development. We use the impossibility of exhaustive testing as an excuse for hardly trying.
But the excuses are thin, because more and more enterprises are shipping software that's adequately reliable and close to on time. They have found the things worth observing in the software-development process and have learned the perils of ignoring relevant measurements. Some of us remember the days of game development for the Atari 2600, and can contrast those wild and wooly times with today's market for embedded and desktop applications. We may have just the beginnings of a science, in Lord Kelvin's world view, but we've still come a long way.
What prompted this essay was a series of gaffes I've personally made in just this area over the past month or so. I'm a guy who writes software for a living, so I should know better. I also preach to others about how to do it right, so I should be even more cautious. But luckily for me, I can still appreciate the education that accompanies most humbling experiences.
For many years, I preached that software development was more of a craft than an engineering discipline. I now believe that is has become an engineering discipline, but has a long way to go before it becomes a science. That's still real progress. Craftspeople learn by observation and by trial and error. They pass on lore to their successors. They are scientists in the early, data gathering stage. Engineers mostly need rules for making accurate predictions, just like scientists in the middle stage of modeling what they've observed.
Scientists look for elegant and robust theories that predict the maximum number of surprising effects from a minimum of assumptions. That's true in some branches of what we call "computer science," but not yet so in the pragmatic world of software development. Maybe it will be some day, but meanwhile we'd all better keep testing.
P.J. Plauger
Senior Contributing Editor
pjp@plauger.com