When I was in college, I invited Francis Corbett, an engineer who had analyzed photographs and home movies of JFK's assassination, to give a talk on image-analysis algorithms to our computer club. As it happened, a bunch of rowdy nonstudents who believed JFK's assassination was a conspiracy showed up to the lecture. Corbett, fortunately, had dealt with hecklers before, and was handling the situation much better than I. Luckily, the lecture was also attended by some exceptionally large and well-mannered folks who quieted down the rowdies so that Corbett could finish his talk.
I learned a couple of important lessons that day -- none of which had anything to do with image processing. The first was to never, ever sponsor a public lecture that has anything to do with JFK's assassination. The second was that physical intimidation can have a remarkable calming effect on people. To this day, I owe those big hairy guys with the leather jackets a debt of gratitude.
It's easy to laugh at conspiracy theorists (unless, of course, they're physically present and outnumber you). But it's hard to deny that for every conspiracy theory, there's often a grain of truth. Most of the time, that grain is insignificant, but sometimes, it's impossible to ignore. And when you strip away the inevitable name-calling and hysterical references to Communists, paparazzi, and Bill Gates, you are left with some facts that can get you thinking.
On February 25, 1997, for instance, the Information Technology Association of America (ITAA) released a report entitled, "Help Wanted: The Information Technology Workforce Gap at the Dawn of a New Century." The message was straightforward: We're running out of skilled technology workers, and this will adversely affect our economy. Our society's growing reliance on software, the report claimed, was creating a greater demand for software professionals. The report went on to state that our educational system was not meeting this demand.
Business journalists at the time must have been bored by all of the positive news about our better-than-ever economy, because the report managed to seep into the headlines of several major newspapers and magazines. Software, after all, had become the nation's third-largest manufacturing industry with wages twice the national average, and the software industry was growing over twice as fast as the rest of the economy. If programmers couldn't meet the demands of the software industry, then the economy was in serious trouble.
If you're a programmer, of course, you're thrilled by the report -- it means higher wages, more incentives, and even societal status. And there's no reason to believe it isn't true. After all, what does the ITAA (or anyone else, for that matter) have to gain from creating a programming shortage out of thin air?
For starters, aliens -- not the kind from outer space, but the kind from other countries. Immigrants tend to accept lower wages than U.S. citizens for equivalent jobs, hence their attractiveness to large ITAA-member companies wanting to increase profit margins. The ITAA report suggests that non-U.S. workers might be one of the solutions to the shored up demand. Indeed, in his paper "Debunking the Myth of a Software Labor Shortage," Norman Matloff, a professor of computer science at the University of California at Davis, cites a San Jose Mercury News article (November 21, 1997) where "the ITAA stated that its 'number one priority' would be to increase the yearly quota of H-1B work visas."
Additionally, Matloff points out that there are indeed engineers who could easily fill the current positions. These are the same engineers who were downsized in the late '80s and early '90s, and who may not even be in the tech industry currently, which is why they wouldn't be accounted for in the supply numbers. However, because older and more-experienced engineers command higher salaries, Matloff argues that companies would rather spend training money on cheaper immigrants and students coming out of college.
There was more fodder for conspiracy theorists in the ITAA study. The report noted that undergraduate computer-science degrees awarded by U.S. universities declined 42 percent between 1986 and 1994. What it didn't say was that the trend reversed itself in 1995, and enrollment was significantly higher in 1996. The Computing Research Association, a national consortium of university computer-science departments that released this study, had, according to Matloff, brought this study to the ITAA's attention before the final version of the report was released. However, these statistics garnered no mention.
The ITAA undoubtedly has its own reasons for hyping its study on the programmer shortage. But it's important to remember that the report also emphasized the need to improve the educational system, to encourage students to pursue a high-tech career, and for greater corporate philanthropy. When ITAA president Harris Miller testified in front of the Senate Committee On Labor And Human Resources in April, 1997, he did not even mention immigration, focusing instead on the need for improved education. And at the ITAA-sponsored National Information Technology Workforce Convocation in January, 1998, topics on the agenda included basic math and science competencies, and recruitment of minority groups -- hiring non-U.S. workers was not on the agenda.
(Curiously, at the same convocation, Matloff was invited to present the programmer shortage issue. Matloff's talk was not on the original agenda, and was placed on the program only at the insistence of the House of Representatives Commerce Committee, who wanted the conference opened to other viewpoints. But the press was noticeably absent from Matloff's talk. The ITAA had bused most of the press across town to attend a press conference scheduled at the same time as Matloff's presentation. Miller called the suggestion that this was an ITAA conspiracy "nutty.")
The reports released by the Stanford Computer Industry Project (SCIP), one of which is published in this issue, may be part of the reason that the proposal for loser immigration laws has not been emphasized in the context of the programmer shortage debate. SCIP, while reinforcing the notion that foreign countries are a good source for temporarily plugging the gap, concluded that, in the long term, the programmer-shortage problem is global. The entire world, not just the U.S., is suffering from a shortage of programmers. While SCIP endorsed using non-U.S. programmers as an immediate solution for meeting demand, it felt the solution was only temporary and limited in its effectiveness.
SCIP addressed other deficiencies of the ITAA report. In a recent New York Times article, Shirley Tessler, one of the coauthors of the SCIP report, argued that the recent upswing in computer-science enrollment was not encouraging. Tessler stated that many of those students would drop out, and many were non-U.S. students who would return to their native countries upon graduation.
Most recently, the Department of Commerce's Office of Technology Policy (OTP) released a report entitled, "America's New Deficit: The Shortage of Information Technology Workers." While not releasing any new data -- it reported the results of several different studies, including the ITAA and SCIP studies -- the OTP was fairly rigorous in its analysis.
The OTP report made the important point that in all of the various studies, the term "IT worker" had different meanings. For example, the ITAA report defined IT workers as those who study, develop, implement, support, or manage computer-based information systems, a mighty broad definition that would include computer scientists, programmers, testers, customer support, system administrators, and others. The OTP report offered four categories of IT workers: computer scientists, computer engineers, systems analysts, and computer programmers. The OTP report also pointed out that most of the studies' projections did not take into account IT workers who did not have a formal computer-science background.
Cries of impending labor shortages in high-tech fields are nothing new. Longtime fans of "The Andy Griffith Show" know that phones didn't always have buttons or even dials. Calls were made by picking up the receiver and asking the operator to call a particular location. While suitable for Mayberry, systems like this weren't very scalable. It was predicted that under this system, there would eventually have to be one telephone operator per person. In 1916, at a Bell Systems Technical Conference, managers discussed the "difficulty of securing and retaining an adequate number of operators." The telephone company was worried that wages were high and the supply of "qualified" operators were low.
Three years later, AT&T adopted automatic switching in large cities. The eventual total adoption of automatic switching essentially converted everybody into his or her own telephone operator, thus providing the necessary supply and averting the predicted communication crisis. As quickly as the cries of paranoia had started, telephone operators were suddenly out of a job.
Similarly, in the early 1960s, projections suggested that everybody would have to become a computer programmer in order to meet the expected demand. The government poured money into educational and training programs in response. Once again, however, advances in technology -- compilers, cheaper and faster computers for the masses, structured and object-oriented programming -- slowed the demand curve to the point where, by the 1980s, people believed that the market for programmers was saturated.
Robert Rivers, chair of the American Engineering Association's Manpower committee, points out that in a market economy, long-term shortages cannot exist. However, this is only true if the supply of workers is elastic.
This is the defining question. Those who believe that the supply of IT workers is elastic believe that, in our free market, rising salaries are enough of an incentive to encourage people to enter the field. Those who believe that the shortage is a real, long-term problem, believe that IT workers are not commodities, that only a special subset of society is qualified to become an IT worker, and that other things are needed to address this impending shortage.
The past seems to indicate that the supply of high-tech workers is elastic, and the demand, cyclical. But Tessler disagrees, stating in a recent EE Times article (April 7, 1997), "This is different. It transcends all industries. I don't see the need for software as cyclical at all. I see strong, steady growth."
The more important question is not why is supply so low, but why is demand so high? Clearly, technology pervades our society more than it ever has. Was this simply a well-defined trend from the beginning that we all just ignored until now?
Embedded-systems development has created some of this demand, but I don't think this the main cause. True, microprocessors are rapidly replacing specialized hardware, but it's likely that engineers can transition from drawing schematics to writing code.
Consumer software companies are not likely to be the problem either. The consumer software industry has proven itself to be extremely fickle over the years -- companies that dominate the market and the public eye often quickly fade away, only to be replaced by new companies -- and there's no reason to believe that this will change.
Most likely, the biggest cause of this growing demand are large corporations spending billions of dollars a year on IT, based largely on the belief that throwing money at technology will improve productivity and profits. Economists and businessmen, such as Thomas Landauer and Paul Strassman have disputed such practices, showing that many organizations have thrown away millions of dollars as a result of this belief. In a wicked twist of irony, the U.S. government recently announced it will spend $28 million to help train and place competent IT workers, of which $8 million will be spent on a web site for job announcements. In other words, the government is devoting nearly a third of its investment to an IT project that, by creating more IT demand, may actually contribute to the very problem it is trying to address. Niccolo Machiavelli was prescient when he observed that one need not look for conspiracy if incompetence can explain most mishaps.
Corporations (and maybe even the government) may eventually wise up, and these and other inefficiencies in the system might go away. Lower and better-managed spending could decrease the current demand for IT workers. But it's impossible to know for sure whether or not new inefficiencies and promises of a better technological future will create new demands to replace these current ones.
What the future holds for IT worker -- and professional programmers, in particular -- is unclear. What is clear is that the demand for programmers is high right now, and the pay is good. If you keep your skills honed and your mind open, you'll be gainfully employed in the future, programmer shortage or not. And that's no conspiracy.
--Eugene Eric Kim
technical editor