Over the last several years, I have written quite a few articles in the Engineering Notebook column of the C++ Report. These articles began with a series on the principles of dependency management. Then I wrote several articles that provided a UML tutorial. Following that was a series of articles about lightweight iterative and incremental processes. And finally I wrote about the most famous of the lightweight processes: XP. (You can download all these articles from the publications section of http://www.objectmentor.com.)
This article is by no means my final installment in the Engineering Notebook column. However, I think the time has come to provide some kind of conclusion to tie the previous articles together.
There can be little doubt that the software industry is in trouble. Software engineers often work hellish hours under severe pressure. The software produced is often buggy and hideously hard to maintain. Frequently the code is so badly written that subsequent changes are very difficult and expensive to make.
To combat these problems, software processes were put in place. These processes often recommend that coding be deferred as long as possible, and that much effort be spent up front on design. The problem is that after months of design the needs of the business have changed, and the resulting code produced is produced with long hours and high pressure and doesn’t look much like the design.
What we need, and what my series of articles has tried to provide, is a way to produce high quality software at a sustainable rate. We need a set of principles, tools, and processes that allow us to develop our software quickly, but without long hours and demonic pressure. We need a way to recognize bad code and repair it before it harms us. We need an accurate way to estimate what needs to be done and a scheme for allowing our customers to use those estimates as input to select the order in which features will be produced. We need a way to get feedback from the customer as early and frequently as possible so that our software never goes too far off track.
Writing software is like running a marathon; it requires a long sustained effort. Yet many developers treat software development as though it were one long sprint. Sprints exhaust us to the extent that we spend more time exhausted and recovering than we spend developing. Developing at a sustainable rate is better. Though not as immediately fast as a sprint, it is faster than sprinting if averaged over a time frame of months. Indeed, if engineers remain rested, they will produce higher quality code in much less time on average than a team that is sprinting and spends its time exhausted.
A rested team can apply the principles of dependency management to keep their code from becoming a nightmare. A sprinting team has no time for such niceties, and an exhausted team has no energy. They will create cluttered and poorly organized code, vastly increasing the difficulty in making later changes or fixing problems. Such code accumulates cruft and dependencies that ruin its structure. The effort required to maintain it increases exponentially as the software grows.
Long, up-front analysis and design phases do not solve this problem. Indeed they make it worse. The time required to do a detailed analysis and design is often long enough such that by the time it is complete, the business environment has changed dramatically. Also, even if the business environment remains constant, the analysis and design is nothing more than guesswork. Customers don’t really know what they need until they see the system working. Thus long, up-front design cycles have a high probability of specifying a system that the customer neither wants nor needs. The time spent producing this specification is lost, and developers must sprint hard to actually build the system.
The problem is solved by developing the software in short iterations and allowing the customer to select which features will be developed in each iteration. The developers estimate how long each feature will take to develop and choose a duration for the iteration. A good duration is usually two to three weeks. The customer chooses features that will fit into the iteration. The customer cannot ask the developers to build more than their estimates say will fit; and the developers cannot ask the customers to change the order of the features.
The developers estimate features in a way that allows them to develop them and then refactor them according to the principles of dependency management. It is critical that the developers keep the code clean and well organized; this effort is built into the estimates that they give the customer. The customer cannot tell the developers to shorten their estimate by skipping the refactoring and application of principles. If the developers are asked to do so, they should refuse. Developers should look at this step the same way doctors look at sterile procedure. It is one of the essential practices that makes software development successful.
As new features are added to the system, the structure of the code will need to be changed. Simply adding new features without refactoring the code leads to a buildup of cruft and dependencies that will eventually make the code unworkable. Thus, developers must continue to apply the principles of object-oriented design and the practice of refactoring according to these principles, throughout the course of development.
This part of development is often called maintenance. It is a common belief that maintenance requires less skill than initial development and that initial development is more challenging than maintenance. After all, initial development is when all the design work is done. During maintenance, we are just making small incremental changes to the code. This view is horribly wrong and has caused more software disasters than any other software misconception I know of. Maintenance is the hardest and most challenging part of software development. During maintenance, the developers must migrate the design of the software to support new features. This migration takes much more insight and skill than coming up with a design without any other constraints.
Therefore, we want to move a software project into maintenance as quickly as possible. We want the initial release to be short and to produce the minimum possible product that is still usable by the customer. We want to release this minimal product to the customer and have him use it and send us feedback. And then we want to continue to flesh out the project in a long maintenance cycle that ends when the product is killed.
This style of development requires that we be able, and unafraid, to make changes to the code at any time. If we keep the code clean by employing the principles of dependency management, then we will be able to make changes when we need to. And if we have a framework of unit tests that verify the proper functioning of the code, then we will be unafraid to make any changes that are needed.
Thus, developers need to include in their estimates the time required to produce the unit tests that verify that the code functions properly. Without these unit tests, major refactorings will be too scary to contemplate. Developers will worry that the changes they make to keep the code clean will break something. This fear will make them reluctant to make structural changes that they know need to be made. Thus, if we are to continually keep the code clean and well organized, we must have a network of unit tests.
To this end, it is best if the unit tests are written before the code. Indeed, as each new chunk of code is contemplated, it should be described by a unit test that fails because the code does not exist. Once the unit test is written, the code that makes it pass can be added to the system. This ensures that every chunk of code has an appropriate unit test. It also ensures that the developers think about the code they are about to write from the point of view of the code that will use it (i.e., the tests). Finally, the unit test is a form of documentation that other developers can read to determine how the code works.
It is imperative that customers and developers alike realize that the value of code is only partially in its behavior. The structure of the code is also valuable. Indeed the structure of the code will have a major impact on whether the product will enjoy long term success.
If the structure of the code is poor, it will become ever more expensive and time consuming to maintain. Developers will become demoralized working with it and will eventually demand that the product be redesigned. Such big redesigns rarely ever result in a superior product. Indeed they are terribly risky and most often take a very long time and produce an inferior result. The reason is often that the new product spends a very long time playing catch-up to the old product, which must continue to be maintained until the new product is ready to replace it.
If the structure of the code is good and is kept clean, then maintaining it will be easy. No big redesign will be necessary, because small redesigns will be taking place continuously. Such a project has the best chance of having a long and profitable lifecycle.
Robert C. Martin has been a software professional since 1970. He is president of Object Mentor Inc., a firm of highly experienced experts that offers high level object-oriented software design consulting, training, and development services to major corporations around the world. In 1995, he authored the best-selling book: Designing Object Oriented C++ Applications using the Booch Method, published by Prentice Hall. In 1997, he was chief editor of the book: Pattern Languages of Program Design 3, published by Addison Wesley. In 2000, he was editor of the book More C++ Gems, published by Cambridge Press. From 1996 to 1998, he was the editor-in-chief of the C++ Report. He has published dozens of articles in various trade journals and is a regular speaker at international conferences and trade shows. He is currently working in the area of lightweight productive processes and is a strong advocate and supporter of Extreme Programming.