Dr. Dobb's Journal August 2002
Progress curves reported at the end of software development projects typically look like Figure 1. Details vary, but the curve's general form is remarkably consistent and reproducible. Referred to as "S-Curves" because they resemble the letter "S," these curves describe a wide variety of human behavior; the classic learning curve, for instance, assumes this shape. Three regions characterize the curve: The flat part early in the project (the first 20 percent or so), the ramp in the middle, and another flat part late in the project. As Table 1 shows, you can associate interesting behaviors with these regions for at least three seemingly different activities.
Why do these seemingly different activities all exhibit S-Curve behavior? What underlying forces produce this curve over and over again? To address these questions, I'll focus on the software development process.
As anyone who has ever worked on a project knows, projects, like physical bodies, have inertia: If you leave them alone, they either stay put or continue to move as before. The measure of inertia is mass — the more massive the body, the harder it is to influence. In this analogy, large, complex projects tend to be more massive than small, simple projects.
To change the state of motion of a body or a project, you must accelerate it. Newton said that the acceleration of a body is proportional to the force applied to it. Without any other compelling theory or data, we assume that Newton's Law applies to projects as well as to physical bodies.
But here's the problem: We experience acceleration when our car peels out at a stoplight, when we stomp on the brakes, or when elevators take off or come to a sharp stop. We rarely measure it, however. More often, we measure two other variables — velocity (speed) and position (location, or where we are). But force is only indirectly related to these variables.
So how are position, velocity, and acceleration related? By definition, velocity is the rate of change of position, and acceleration is the rate of change of velocity. If you plot the position of a body versus time, you can get the velocity by looking at the rate of change of the position. Similarly, once you have the velocity curve versus time, you can look at its rate of change and obtain the acceleration curve versus time.
In calculus, this means taking the derivative of the position function to get the velocity, and taking the derivative of the velocity function to obtain the acceleration. The acceleration is, therefore, the second derivative of the position. Even if you don't know these functions analytically, you can always take the derivatives numerically, so long as you have position data.
According to Newton, acceleration is proportional to the force applied, so the second derivative of the position of an object is proportional to the force.
Remember: You can find the derivative (rate of change) for any curve by looking at a line tangent to the curve at a given point. The value of the tangent's slope is equal to the derivative at that point. But can you use reasoning of this type to understand the forces at work on software development projects?
Again, most projects can be plotted according to Figure 1, a "position versus time" curve. What does the velocity curve look like?
Taking the rate of change of the project completion curve in Figure 1 yields Figure 2, in which the vertical axis has been normalized. This graph tells you that projects start off slowly, move faster until they reach maximum velocity (the halfway point in Figure 2 because the project completion curve is symmetric), then start slowing down. As you cross the finish line, you are actually going fairly slowly, much as you did at the beginning.
This is consistent with everyday experience. Finishing is hard. Getting all the fine details worked out so that you can deliver the product seems to take forever. Often it feels like you stagger across the finish line. So what does this say about the underlying forces driving the project? Well, at this point physicists would take another derivative.
The resulting acceleration curve, which implies the forces at work, looks like Figure 3. During the first third of the project, you see an increasing positive force. This corresponds to a lot of enthusiasm for the new project, the addition of new team members, and general optimism about the chances of success. Cynics might call this the "ignorance is bliss" period, but that positive, increasing force causes the project to gain velocity.
At about the one-third point, this positive force begins to decline. Reality is setting in; people are far enough along to start to understand what the real problems are, and beginning to feel some schedule pressure. After all, they have now used up one third of the time but still have mountains of work to do. Also, the team is beginning to feel the full burden of a large staff; a lot of time is spent in meetings, and communicating information to all team members grows difficult.
At the halfway point, according to Figure 3, a negative force sets in. The team begins to feel the project pushing back. Many really tough problems are not succumbing to solutions as quickly as thought. People begin to panic as more and more sand slips through the narrows of the hourglass. At about the two-thirds point, the force hits its maximum negative value — there is this sensation of swimming in molasses. If the project stays here for long, it dies.
And then, when you most need it, there is a breakthrough. All of a sudden, things don't look quite so bleak; although a negative force is still at work — you know there are a million details left to complete and not much time to do it in — this force decreases. The team can see the finish line, and the negativity decreases until you cross it.
Be aware that the actual turning points on this curve — one third, one half, two thirds — vary from project to project, and in turn affect the velocity curve and project percent completion curve. This is gratifying. You started with a prototypical percent complete curve, took derivatives, and inferred the forces underlying the project. Empirically, the data seems to fit the theory. But have you really accomplished anything here?
What you would like to be able to do is predict the percent complete curve while in a state of incompletion. In other words, you want to know when you will be finished. At any point in time, all you have is that portion of the curve behind you, and some notion of the velocity curve. The forces curve is hard to pin down quantitatively. In fact, it is often hard to understand the velocity curve, and metrics that could indicate velocity would be very, very useful. Today, many project metrics focus exclusively on position — "Where are we?" To forecast accurately, however, you need both actual position and actual velocity.
Should you ever have the ability to understand how the velocity is changing, then you have a more complete picture. So you need to think about project metrics collection in this light.
All this is somewhat introductory; you don't do software development projects in one fell swoop. Instead, you break them down into iterations — each with its own rhythm — so Figures 1 through 3 are really only approximations. For example, for a project with four iterations, the percent complete curve would probably look more like Figure 4, where four S-Curves are stacked on top of each other. Percent complete is cumulative, and you assume that each iteration takes 25 percent of the time and gets you 25 percent of the way along on the completion curve.
Now jump to the velocity and acceleration (forces) curves that correspond to this four-iteration percent complete graph (Figure 5). What do these graphs of the derivatives tell us?
First, you see that the velocity curve replicates itself four times — no surprise there. The project gains and loses velocity each time, as each iteration follows a somewhat similar rhythm. The second derivative curve — acceleration — has three discontinuities. At the start of each iteration after the first, you need to kick out the residual negative force from the previous iteration and impart an initial positive force. This is what causes the velocity vector to change abruptly. Without this instantaneous, discontinuous force, you can't get going again without losing time.
You shouldn't be surprised by the discontinuity — after all, you started at zero with a nonzero positive force to get the project underway. Most good project managers recognize this need and apply requisite force at the appropriate time. After applying this positive force, you need to once again build it until the iteration hits its wall, and the inevitable negative forces set in. Then it's just a question of hitting the breakthrough for that iteration, the one that again reverses the negative force curve.
In real life iterative development, there are many iterations, divided into distinct phases. How can you inject this sort of reality into this model?
The Rational Unified Process, for instance, defines four phases: Inception, Elaboration, Construction, and Transition, each with different characteristics. Early in the project, you are doing a lot of discovery (new learning). In the middle, you do somewhat less discovery but lots of invention; there is still a lot of learning going on. Later, you are actually doing activities that count for completion: some invention and lots of implementation. Learning drops off as the project moves toward completion.
It's not clear that you need the complexity of a multiphase, multiple-iterations-per-phase graph. After all, it is the phases that differ most in their characteristics, not the iterations within the phases. So, to keep things reasonably simple, model the phases and assume only one iteration per phase.
At this point you need to decouple the ideas of learning and completion. Table 2 lists some numbers Philippe Kruchten, author of The Rational Unified Process: An Introduction, Second Edition (Addison-Wesley, 2000), believes are applicable to the four phases. What these numbers tell you is that when you do phased, iterative development, you learn faster than you achieve completion. This accelerated learning is what helps reduce risk. How do you introduce these ideas into your model?
As Figure 6 illustrates, you only need to make two modifications:
First, let's address the different shapes of the learning and completion curves. Basically, you achieve 60 percent of your learning in the first 40 percent of the project, but you are only 25 percent complete at that point. This reflects the notion that in iterative development you emphasize learning early to reduce risk. The counterweight is that you don't make much visible or tangible progress because often the learning has few associated artifacts.
Now look at the velocity curves. Learning velocity peaks during Elaboration, whereas completion velocity peaks during Construction. This is consistent with our model, in which you set out to get over the learning hump in Elaboration, sacrificing or deferring completion a bit. The measurable artifacts of completion begin to show up during Construction, so that is when that curve peaks.
You might ask why both velocity curves go down to almost zero at the 40 percent point, then back up. Wouldn't it be better if you could keep some of that velocity across the boundary? Yes, it would. Distinct phases and iterations within phases cause disruption, which is why many senior project managers try to blend things at the boundary; they sometimes get an advance team working on the next phase or iteration before the antecedent phase is actually done, to smooth the transition over the boundary. It is hard to pull this off. The more iterations you have within phases, the more boundaries you create. If there is a fixed overhead at each boundary, then you increase your total overhead by increasing the number of iterations.
The forces comparison for Elaboration and Construction is also pretty clear: The learning forces are stronger during Elaboration and relatively weak during Construction. On the other hand, the completion forces have about the same peak values during both Elaboration and Construction, which is a good thing.
Now address the Inception and Transition phases. You have large forces at the end of the project, during Transition. That's what makes finishing so hard — you need a big push when the team is the most tired. And there are also large forces early in the project, during Inception, because at the end of Inception you face your first "go/no go" decision. If the project team is concerned about cancellation at this early branch point, then you can be sure these large forces will be at work.
Assume that you can add a project's completion forces and learning forces with the total force as a function of elapsed time. As Figure 7 illustrates, all the segments under the curves in the four regions have the same form, so the product of the peak value of the total force and the length of time the segment is in play is proportional to the area.
We know that the area is an integral, and the integral of force over time is equal to the change in momentum. Table 3 shows that the change in momentum (or impulse) is roughly the same over the four phases. It's the same during Construction and Transition — 10 units. The biggest impulse is during Elaboration, 12 units. And the combined impulse of the first and second phases — 20 units — is equal to the combined impulse of the third and fourth phases. Half the total impulse is applied during the first 40 percent of the elapsed time; this again expresses the idea of front loading, which we believe is a good thing.
The conceptual model presented here can help prepare you to deal with the various project phases as they play out. Knowing that some degree of determinism is at work can be comforting when a project hits its low point and you're worried that the team may lose hope.
DDJ