Dr. Dobb's Journal April 2004
When I earned my living doing Operations Research (an early name for what became "Systems Analysis"), we had two major problems—the classical OR problem of modeling the activity or process we were studying, and the far more difficult problem of solving the model. The University of Washington Grade Prediction Program, for example, needed to manipulate about 60 variables and tens of thousands of student records. Building a linear regression equation required the inversion of a 60×60 matrix—something no computer of any size had ever done.
Some of the most complex models we could then devise can be solved today in minutes to hours on desktop machines. That's the good news. The bad news is that a lot of the old models we thought would be pretty good if only we could solve them turned out to be pretty bad. And meanwhile, a number of factors changed in the real world. For instance, at one time, the ratio of the squares of the forces engaged was not an unreasonable first-order approximation of the results of a battle. Manipulating those could give a rough estimate of the casualties each side would take. Now think of a tank battle in which one side can engage in foul weather and dust at a distance of four miles with a single shot probability of kill well above 0.5, while the other side can't even engage until the distance closes to under a mile. Then add in communications discrepancies. Finally, add in any kind of morale factor.
The result will be a one-sided victory well outside anything in modern experience (until quite recently), with one side's casualties slightly exceeding its natural attrition rate due to accidents. On the other side, casualties will approach 100 percent—only we don't need any sophisticated analytical techniques to predict that result, and the challenges to the prediction depend on surprises and complete unknowns.
An actual Monte Carlo model, in which you keep track of every unit engaged and assign various probabilities for each event—probability that the observation systems will see the enemy, probability that the communications from the observation system to the command planning center will work, information transmission rates, probability that the orders will reach the units engaged, probability that the unit will be ready to fire, weapon CEP against target at that range, and so on—can be made so complex that you can't solve it. Still, it would give about the same result—one side's casualties will be on the high side of the accident rate in that terrain for that deployment, and the other side's will approach 100 percent. Refining the model to exceed the capability of the computer, then getting a better computer and running that refined model, won't change the prediction by much.
On the other hand, we still have models that already exceed the capability of existing hardware to solve them. Climate models come to mind: Computer models predict all kinds of disastrous trends that no one, so far, finds in the actual climate data. Since the models don't do a good job of predicting the present from the past, it's easy enough to understand the reluctance to spend huge sums on remedies—reducing CO2 emissions comes to mind, a terribly costly undertaking particularly for developing countries—until there's more evidence that the remedies are actually needed and will do some good.
I'm not sure Moore's Law will operate to get us past that particular problem; the computational power needed to simulate world climate is many orders of magnitude above what we have now, and that's assuming the world climate system can be modeled at all. An order of magnitude is a fancy way of saying times ten; it takes about six years of Moore's Law to produce a ten-fold improvement, or an increase in capability of 1000 in about 20 years. Moore's Law doesn't strictly apply to supercomputers, but it's a good rule of thumb; and it's not clear that even a factor of 1000 improvements in computer capability will give a good enough simulation of world climate. By "good enough," I mean a simulation that—given what we knew in 1900—would predict the climate of the year 2000.
Conversely, we already have more than enough computing power to determine whether the payoff is higher for spending on decreasing uncertainty or for undertaking remedies despite uncertainties. This is one form of classical Bayesian analysis. The inputs are costs of various remedies including the incremental cost of starting a remedy later rather than now; the probability that a given remedy will be useless; and the information costs of reducing that uncertainty. When you apply that to climate and global warming, the answer is always the same—reducing uncertainty. That is, getting more information on what is really happening is a higher payoff strategy than spending money on remedies that may not be needed. Note also that the remedies will probably cost less in the future than they do now, assuming they are needed, because you will start from a higher industrial base.
Costs of information acquisition are also falling—sensors and probes and data storage systems get cheaper along with processing power. This makes the case for finding out what we ought to do before spending a lot of money on doing it even more compelling, and is very likely the basis for the Russian decision not to ratify the Kyoto treaty. The Russians have always been pretty good at mathematical analysis even with bad computers, and high-capability computing at commodity prices helps them a lot.
One unexpected consequence of the increasing availability of computer power was the Bowl Championship Series (BCS) mess. When I was an undergraduate, there was only one really important Bowl Game (assuming you were in the Big Ten or the Pacific Coast Conference), and that was the Tournament of Roses, otherwise known as the Rose Bowl. Other postseason games wanted in on the act, and somewhere along the line came pressure to have a single Bowl game decide which college football team was number one for the year.
Since college presidents absolutely rejected the notion of a playoff series on the grounds that this would seriously interfere with academic excellence, who got to play in each bowl was changed (the Rose Bowl was no longer the top Big Ten versus the top Pacific Conference team, and so on), and the game designated as the title game rotated among several bowls including the Sugar Bowl in New Orleans and Rose Bowl in Pasadena. The winner of the game designated as the BCS Championship Game would then automatically become Number One.
The top ten teams were chosen by polls: designated coaches and sports reporters. So long as these picked the same teams as either #1 or #2 all was well, but when that didn't happen, there were problems choosing who got to play in which bowl; and while some Bowls always sell out, some might not unless hosting the BCS Championship Game that year, so there was a potential for considerable money to be at stake. And, inevitably, there came a year when the two polling groups disagreed, and there were two "champions" and everyone was upset.
Along came a bunch of people with computer models, which they promised would do a better job of choosing which teams were "really" the Number One and Number Two teams. One computer model designer, a professor of political science at the U.S. Air Force Academy, has even written articles claiming that the computer models are "more just" than the polls of coaches and sports writers—in other words, the computers do a better "expert" job than the experts. Eventually, seven computer models were chosen by a process that I have been unable to discover. None of these models is public. Some tell you what decision factors they consider, but none of them publish the actual model including the weights given to each of the factors. Needless to say, this means that whatever else the model is doing, it is not making the decision process explicit and objective, despite the claim that these models are more "objective" and "just" than polls of experts.
All this would be of little concern to anyone not seriously interested in college football, except that it is illustrative of something far more important—just what are computer models for?
A long time ago, Herman Kahn and others took the techniques known as "Operations Research" and fashioned something called "Systems Analysis," which was a fancy term to describe a computer model for making decisions and choosing strategies. Kahn, in his classic Techniques of Systems Analysis (Report, Rand Corp., 1956-1957. U161.4K33.), made it clear that the point of all this was to make decision factors as explicit as possible, and give everyone some idea of the weights assigned to each of those decision factors—in other words, to try to quantify "gut feelings."
This was greatly resisted by some military commanders, who felt their judgment was being called into question. Kahn's reply was that it couldn't hurt to know as much as possible about why certain decisions were made the way they were, and this was no attempt to take the decision power away from the commander, but rather to help him make up his mind. That controversy raged throughout the services in the late 1950s during the reign of Robert McNamara at the U.S. Department of Defense. McNamara claimed that Systems Analysis was better than judgment, and gave objective grounds for decisions; commanders said this was a way to avoid taking responsibility for decisions made. Analysts such as Kahn and Francis X. Kane not only agreed, but spoke of the "figure of merit fallacy"—the assumption that all decision factors could be combined into a single figure of merit that would automatically choose the best strategy. Possony and I dealt with some of those matters in The Strategy of Technology (University Press of Cambridge, 1970; http://www.jerrypournelle.com/sot/sot_intro.htm).
In those days, the controversy was tempered by two factors: The computers available weren't powerful enough to solve most of the complex models the systems analysts came up with. Moreover, McNamara had a terrible habit of fudging his data to make the results come out the way he wanted. Eleven military boards chose Boeing to build the TFX (later the F-111), but somehow McNamara's whiz kids came up with a way to show they were all wrong, and the contract went to a Texas company, so that the TFX was known in the industry as "the LBJ." In no case were the computer models that chose the winner made explicit. "That's what the computer decided" was deemed sufficient justification.
All that's history. But the BCS controversy shows the issue isn't dead, and the figure of merit fallacy has infected the political science department of one of our service academies.
Blind faith in "computer models" is stupid. It is time for those who use them to understand the purpose of such things, which is to make explicit the factors considered and the weights given them.
Having said all that, it's important to note that increased computing power does make it possible for decisions to be more objective, precisely because the decision factors are out in the open. Moreover, with the computer power available on the average desktop, it's possible to make highly complex linear regression models that will do a pretty good job of predicting anything regularly predictable.
My favorite example is the University of Washington Grade Prediction Program devised by Professor Paul Horst of the Department of Psychology working under a contract from the Navy Electronics Laboratory. This was a pure statistical dragnet: Take every predictor you can devise including high-school class standings, SAT scores, IQ, high school GPA, estimates of the academic standing of the high school graduated from, and anything else you can imagine might be related to success in college. Put those into a predictor matrix and wait. When the student graduates, you have a GPA in a particular major. That's what is to be predicted.
Given enough students, you will eventually have enough data to generate (through rather simple matrix algebra calculations) a formula combining your predictors to generate a vector of predicted results: predictions of grades in various majors. You now collect all the significant predictor data on incoming students, run the program, and give the result to the student as an aid for deciding what major to choose. Note that this is entirely "objective." The prediction factors are chosen by their predictive value, not by any prejudice of the experimenter; moreover, new prediction factors can be added and tested as you add succeeding graduating classes to the data pool and refine the model.
This worked quite well, and the technique is applicable to many predictions. In the 1960s, I used a modification of this technique to generate a model predicting the future sales of "O" rings of various sizes, using mostly past sales data but also projected growth rates of companies purchasing these items, and some other "predictors" that the company sales people thought relevant. (Some of those were, some were not.) The matrix manipulations including inversions were done at UCLA on the best IBM computer available at the time, and the results proved fairly valuable to the company sponsoring the study. Today, such things can be done on a good desktop computer.
The Movie of the Year for 2003 (and well into this year) was Lord of the Rings: The Return of the King. I've seen it three times now and all I can say is "Goshwowoboyoboy" and make other appreciative noises. I only wish Tolkien had lived to see how well they did this. The book of the month is John Keegan's Intelligence in War (Knopf, 2003), which is a good companion to last month's Intelligence Wars by Thomas Powers. If I seem to be stuck on the intelligence game lately, it's because the subject is terribly important. The U.S. has been floundering about due to disorganization in both the CIA and FBI, and those organizations are too important to be left broken.
The Game of the Year for 2003 was Medieval: Total War, from Activision; and yes, I know, it was a 2002 Best of the Year winner in many surveys. I choose games for awards depending on when I play them, not when they were published.
All in all, Moore's Law has given us a heck of a ride, and it's not over yet.
DDJ