Delivering bug-free, truly useful software on time is the holy grail of software development. Unfortunately, most software shops are run in such a way as to make that impossible. Most projects are characterized by lack of design, unrealistic schedules, and poor (or nonexistent) plans for how to develop and test the software. The software-development processes in a company are often chaotic (with no two developers working with the same tools or in the same way). Programs are not really designed. Projects that succeed -- and most don't -- do so because of the heroic efforts of one or two individuals. The Software Engineering Institute at Carnegie Melon (SEI) has come up with a system for identifying a company's ability to produce high-quality software called the "Capability Maturity Model." The behavior I just described is characterized by SEI as Level 1 behavior, and probably 95 percent (or more) of the software companies in the world operate at this level.
Even companies that want to "do it right" often don't. I just did a small consulting project for a company that had gotten about a quarter of the way into the design of a product, then started coding. The developers had all the best intentions, and actually thought that the design was done. But since it wasn't, the coding was not going nearly as fast as it should have. They thought that the problem was some flaw in the design that they couldn't see, but they were wrong. Though I had some quibbles with the design proper, the real problem was that the design just wasn't complete.
I helped them primarily by identifying the parts of the design that weren't done and showing them how to complete it. But the fact that they started coding too early had indeed impacted their schedule, which had already slipped once. (Not finishing a design is common in my experience, by the way. The myth of "analysis paralysis" is just that. I'm sure that a small number of projects have failed because the programmers didn't know when to stop the design process, but the vast majority fail because the programs either aren't designed at all or because the design process is halted too early.)
This situation exists largely because of myths that have proven false many times. Nonsense like "It's impossible to estimate the amount of time it will take to do a software project" and "Programmers work better under pressure" all contribute to programs not working at all. Unfortunately, the studies that have refuted these ideas tend to be buried in obscure journals and books that only academics read. Moreover, this sort of software practice is rarely taught in universities. Software engineering classes, if offered at all, are usually electives.
Steve McConnell, the author of the book I'll discuss momentarily, says it nicely:
Software developers are caught on the horns of a dilemma. One horn of the dilemma is that developers are working too hard to have time to learn about effective practices that can solve most development-time problems; the other horn is that they won't get the time until they do learn more about [these practices].
To the rescue, then, comes a great book: Steve McConnell's Rapid Development (Microsoft Press, 1996). The point of the book is to show you away out of the SEI Level-1 morass. To quote again:
...watching most organizations attempt to avoid these classic mistakes seems like watching reruns of Gilligan's Island. At the beginning of each episode, Gilligan, the Skipper, or the Professor comes up with a cockamamie scheme to get off the island. The scheme seems as though it's going to work for a while, but as the episode unfolds, something goes wrong, and by the end of the episode the castaways find themselves right back where they started -- stuck on the island.
McConnell provides a message in a bottle that, if followed by the various Gilligans out there, might get them off their respective islands at last.
"Come on," you might say, "a book called Rapid Development, written by someone who's worked at Microsoft, published by Microsoft Press -- how could this possibly be worth reading? Microsoft is a classic SEI-Level-1 organization -- a case study in how not to develop software. They pull it off only because they've got a budget larger than the GNP of most industrialized nations and have no qualms about burning people out and throwing them away." In practice, McConnell uses his experiences at Microsoft primarily as an example of how not to do things, and in all fairness, he does acknowledge when something was done well. This book does much to legitimize Microsoft Press as something other than a house organ. The author's earlier Microsoft Press book (Code Complete) was also pretty good. (McConnell also edits IEEE Software's wonderful "Best Practices" column, which covers much of the ground found in this book.)
Rapid Development is an overview of software-development practices that, if actually performed, will increase the odds of delivering a bug-free program on time. McConnell takes the stand, with which I wholeheartedly agree, that software development can be accelerated only if you take the time to do it right. Rapid Development is a systematic discussion of what "doing it right" actually means.
The phrase "rapid development" is meant literally -- how to develop a program as rapidly as possible. The phrase is not meant to describe a pat set of rules and has nothing whatsoever to do with Rapid Application Development (RAD), a term now abused to the point of meaninglessness. McConnell correctly insists that developing good software at a fast pace requires a panoply of techniques used in concert, not one magic quick fix, and he tries to present as many of these techniques as possible. He doesn't present a single patented method: Rather, he discusses various alternatives along with pros and cons.
For example, it's a fact that it takes dramatically less time to do a design and then code than it does to just leap into the coding. I've experienced this in my own work, and several studies have proven it to be so, but many managers still believe that they "don't have time to design." Not doing the design, then, is a factor that slows a schedule. McConnell addresses matters like this one head-on, with hard data and references to case studies that prove his point.
Part One of the book lays out fundamental concepts, such as general development strategies and risk management. A valuable chapter enumerates 36 classic mistakes that are made on typical software projects. (They are each discussed elsewhere in the book as well, but it's nice to get them all in one place.)
The heart of the book goes over the core techniques for speeding up software development: Lifecycle Planning, Estimation, Scheduling, Customer-Oriented Development, Motivation, Teamwork, Team Structure, Feature-Set Control, Productivity Tools, and Project Recovery. This section leads you from the early stages of developing a reliable estimate through the actual release. I found the chapters on Estimation and Scheduling particularly valuable, since bad estimates are often the root causes of other problems.
The chapter on Productivity Tools is also quite valuable since it addresses a problem that I've also seen a lot in the real world: "If only I had the proper tool, I could increase productivity tenfold or more," is a common refrain. Snake-oil products like Visual Basic and Delphi are sold by the millions to people who believe this nonsense.
McConnell refutes this idea by summarizing Fred Brooks in his classic essay, "No Silver Bullet," (reprinted in The Mythical Man-Month, anniversary edition, Addison-Wesley, 1995 -- a book every programmer should read). Brooks contends, and my experience has proven him correct, that productivity enhancement of this sort is impossible. His reasoning is that most of the work done in putting together a computer program is thinking, not coding. Whether you do the thinking up front as a design, or scatter the thinking through the coding process (perhaps the least effective way to do it), at most you'll spend about 25 percent of your time actually coding. Even if a RAD tool could eliminate 100 percent of the coding, the best you can hope for is about a 25 percent improvement in productivity, certainly not tenfold.
I recently worked with a client who fell into this trap. He was looking for tools before he even knew what he was going to build, since no design existed. He didn't even have a full-blown requirements document. My client was willing to distort the design to force it into the mold required by the tool. To make matters worse, his own developers didn't want to take this silver-bullet approach, but my client was terrified by an absolutely inflexible deadline set by the marketing department (not based on well-established estimation techniques -- McConnell discusses this problem in depth as well). My client honestly believed that doing a design before coding was a risk.
The final chapter in this section tells you how to salvage an out-of-control project on the brink of catastrophic failure -- a valuable chapter considering the state of most projects.
The book finishes with a 200-page section describing 27 "best practices," techniques that are all almost guaranteed to increase the efficiency of a software shop. This part of the book is a how-to guide for setting up a high-performance, quality-software shop.
Steve McConnell's Rapid Development is a great distillation of current thinking in software engineering. There's not much original thinking here, but the book collects a tremendous amount of useful material into a manageable volume that a busy programmer or manager can actually find time to read. McConnell writes from an in-the-trenches perspective, and you get a feeling that he writes so convincingly because he's experienced all the problems he discusses first hand.
The book is long -- more than 650 pages, and unlike most Microsoft Press books, contains very little fluff -- but each chapter is of manageable length and stands alone quite nicely. You don't have to read the book in one sitting. The main weakness of Rapid Development is that McConnell doesn't have space to give most of his topics adequate coverage. He makes up for this by providing copious hard references to his sources, so you can look something up to study a topic in greater depth. Hokey but effective little pictures in the margin nicely highlight the data that backs his claims. One improvement I'd like to see is a replacement of the made-up case studies that illustrate the points made in every chapter with real case studies involving real programs. It's not as if there aren't enough real examples of failed projects out there (though maybe there's a dearth of written-up successful ones). I'd also like to see the design process actually discussed. The book just assumes that you already know how to create a good design. A thorough discussion of design principles would, of course, be a book of its own, but I'd like to see a chapter summarizing good design practice.
Consider this book ammunition. For example, all programmers know, deep down, that halving a schedule that is already optimistic isn't going to do anything but make people work more slowly. Nobody's going to bust his butt (to use a technical term) to accomplish the impossible. McConnell has an entire chapter dedicated to just this subject. Armed with McConnell's work, you have a much-better chance of convincing an out-of-control manager that a two-month schedule for a six-month project isn't a great idea. McConnell's Rapid Development should be required reading for anyone who's managing a software project and for everyone who's convinced that they work for the same company as Dilbert.
DDJ