Dr. Dobb's Journal Spring 1998
Projections from all sources show a widening gap between supply and demand for software professionals. This shortage is worldwide, not limited to any specific technology (such as Java, SAP, or Year 2000 workers), and extends across all industries -- not just high-tech.
A number of people we've talked to about the current shortage in skilled software labor argue that, while there may be short-term tightness in labor markets, these bottlenecks can't last very long in a free economy: Salary adjustments and other market forces will naturally balance the system by attracting more people into the field. Indeed, many people point to historical oversupplies of some scientific and engineering disciplines as a caution against pleas to introduce programs designed to increase the number of software developers. While we can't predict the future, we believe that a sustained software talent shortage would have serious ramifications.
Everyone agrees that there are only three ways this work-force shortage can end:
Economists, technologists, and other optimists are certain that either solution #1 or solution #2 will solve the talent shortage problem in short order. In our opinion, damage is already being done: Serious project risks presented by prolonged talent shortages lasting a decade or more will eventually decrease demand for new systems to levels that can be reliably produced by existing talent supplies, which will continue to grow slowly.
The shortage of software talent is caused, fundamentally, by increasing demand for software of all kinds: products from software publishers such as Microsoft, Oracle, and Sega (growing at 12-15 percent each year), corporate information systems from the back office to e-commerce (growing at 10-12 percent), and the embedded software in everything from automobiles to cellular phones to toys (for which there is no data on current size or growth). The broad use of increasingly complex software across so many industry sectors is both the reason for the shortage and the cause for concern -- software has become critical to the economy.
Technologists and policymakers point to erroneous past projections of labor shortages in new technologies, such as the projection that telephone use would grow so large that every person in the U.S. would need to be employed as a telephone operator. They use the telephone operator metaphor to argue that some imminent technical breakthrough will surely reduce the need for software specialists. While it is probable that new software development technologies will eventually appear, there are currently no new tools or techniques on the horizon that will significantly impact the productivity of software developers across broad classes of software. Furthermore, technology adoption is slow enough that even an easy-to-implement and integrate invention takes many years to penetrate even a small portion of the software industry. So, we are still left with a straightforward supply and demand situation for software.
We cannot be specific about what kind of software workers are most critically undersupplied. We believe that there is less of a problem finding entry-level programmers, but that the talent involved in successful software projects cannot always be found in entry-level recruits. Successful software development teams include systems analysts, systems architects, several types of designers, programmers of all sorts, testers, documentation writers, and even trainers. More importantly, successful projects require talent in project management and a number of other technical management specialties. We know that for all skills, the most talented are much more productive and less error prone than the average (a factor of 20 is often cited), and that experience and familiarity with the systems and types of systems they are working on increases productivity for all levels of talent in any job. But we don't know where the talent shortage is most critical, what skills are in most demand, or how entry-level workers match up with the talent levels and specific skills in short supply.
Some economists and government policy makers feel that increasing the supply of software people, by raising salaries and funding new educational programs, is the way to deal with the situation. After all, they argue, many current programmers had no formal training in computing and migrated to the profession from engineering or business or other fields. But software development has changed over the years: It is as complicated as any engineering task, and most people with a talent for software are already doing it. Moreover, demand is growing fast, perhaps two hundred thousand unfilled new jobs a year in the U.S.; and increasing the supply of skilled workers is a slow process. Before significant progress can be made in expanding the trained labor force, many open positions will be filled with inexperienced people who have, on the average, lower productivity and possibly less training or talent in the various skill areas.
This "deskilling" of the software talent pool will eventually slow down the demand for software. Reduced demand seems like an unlikely prediction at a time when everyone owns a computer, every household device and toy has a chip in it, and corporations compete on the basis of Internet marketing schemes, supply chain management, and database marketing. But as software talent becomes scarce, systems built by less-experienced or less-trained teams may result in more- serious losses for many organizations than they already experience -- delayed projects (like airports), abandoned projects (like the IRS revamp), and poor-quality systems (like air-traffic control). Disenchantment and pragmatic reassessment of how many systems their people can realistically deploy will make CEOs, CIOs, and product designers rethink their investments in new projects. This situation is bad news for the software industry and for high-tech generally, but because so much of our national economic advantage is based on the rate of innovation we can achieve, slowing down the software race is also bad for the economy generally.
Of course, if we could quickly increase the supply of talent, we could eliminate this labor shortage in short order. Indeed, if anyone off the street could become a programmer tomorrow, there would be no shortage (and no welfare system). How much can the talent pool be expanded by education, immigration, and cross-over of intelligent people from other fields into software?
We believe that demand is growing much faster than the educational capacity of current institutions, which are disinclined to rapidly grow their capacity. (We have recommended the establishment of new, industry-funded trade schools, but see no likelihood that this will happen in the next decade.) Immigration in the current political climate is not a solution, and, in any case, the shortage is global.
Since entry-level people seem to be easier to find, some believe that software shops that insist on certain specific skills or experience levels are being too selective in their hiring practices. But talented programmers are much more productive than average developers, and they are rare. Furthermore, companies are reluctant to assign inexperienced people to mission-critical projects -- they can slow things down and are more likely to make costly errors.
So it boils down to how many people already employed elsewhere will be attracted into software and how those people who do cross over (because salaries are rising, for instance) will impact the talent pool. Will they be, on average, as productive as the current work force? Not everyone has an aptitude for the various skills required in software: architecture, design, coding, debugging, project management, and so on. Developers at the low end of the talent curve can have a negative impact on team productivity and software quality.
Even with substantial salary increases, software jobs will not attract an adequate number of people from other disciplines with real talent for the various development tasks because:
We are left with a situation in which the labor pool is expanding much more slowly than demand is growing. Firms are trying to find the talent they need for current projects by recruiting it away from other firms. To understand how this dynamic is evolving, it is helpful to segment the industry.
Software is used in many ways to create business value. In some sectors of the software industry, market forces are clearly already at work. Rising salaries are attracting top talent to meet rising production demand: talent from other sectors, from abroad, and from schools. At the top tier of software organizations are the venture capital funded software startups, the "boutique" software services firms, and the software publishing houses (see Figure 1). In this segment, where growth is nearly 15 percent, the operations of natural labor market forces is quite apparent. There are three things to note about this tier of the software industry:
Let's call these firms Tier 1 of the software industry. We believe this segment represents less that 10 percent of employment of professional software developers and produces only a fraction of the total business value attributable to software. Since growth in this tier has for several years involved migration of top talent from the rest of the software industry, we have found an increased awareness of the shortage elsewhere.
We divide the rest of the industry into two more tiers where the labor situation is quite different and where labor market forces have been much slower to operate. In Tier 2, which includes computer and other high-tech equipment manufacturers, communications companies and many firms in financial services and other information-technology-intensive industries, there is a growing realization that software development is critical to competitiveness. Some larger information technology (IT) services firms would be classified as Tier 1, others would have to fall in Tier 2. In Tier 3, which includes most manufacturing and government organizations, people who have realized the strategic importance of software have no choice but to leave -- there is no executive-level awareness here about the key role of software and no chance of responding to labor market changes even if the software's value were clearly recognized.
Since there is no ability or will to deal with the talent shortage in Tier 3, and the Tier 1 firms have mostly figured out what they need to do to recruit and retain top talent, all the market dynamics take place in Tier 2. The firms in this category are beginning to realize the critical role of software in their industries, but are still hobbled by old attitudes, practices, and organizational structures. Their awareness of software's role in their business has moved beyond blaming software for delays in product launches and firing managers for failure to field new information systems. They realize that their products compete on the basis of software-enabled functionality and that their sustainable advantage in manufacturing, marketing, distribution, or sales is increasingly dependent on systems that require advanced software capabilities. However, these firms are unable to respond to labor market tightness as fluidly as those in Tier 1 because:
As a result of these outdated attitudes and management practices, firms in Tier 2, in which the talent shortage has been apparent for many years, are still losing talent, still not raising salaries and changing working conditions adequately to recruit against Tier 1, are still indiscriminate about whom they hire, and still failing to make the decisions required to turn this situation around in order to retain the talent they have and recruit the talent they need.
In each industry, those few firms who use this human resource well will prevail. The others will eventually stop investing, frustrated with their inability to field effective IT solutions and initiatives. The results for high-technology firms and for the economy in general will be serious. The results for American programmers depend on where they are standing when the music stops.
Although it appears that there is no quick solution to the software talent shortage, there are concrete steps we can take to lessen the impact. One approach, of course, is to continue educating and retraining software people. We can also continue tapping the talent pool in other countries (although it is clear that there will not soon be another India). New software-development technologies (such as component-based software assembly) will help too. Employers will have to rethink how they approach the demand for software products and applications, what the necessary skills and job classifications are, and how they train their labor force.
One positive step that employers needing programmers could take is to establish industry-funded training (and retraining) programs in partnerships with schools. Industry might also begin more meaningful certification of some types of professionals. And clearly, research on software-development skills and technologies needs to continue.
Still, we are facing what appears to be a ten-year drought in the supply of software developers -- and things will continue to get worse before they get better.
DDJ
Studies such as a 1997 ITAA report estimate 190,000 open positions for software developers (excluding government and nonprofit organizations). The Gartner Group, on the other hand, is more pessimistic, estimating the shortage at 500,000 people. One 1994 study suggested that the supply of computer-science graduates in the U.S. is declining; see Figure 2.
By most estimates, there are about two million people developing software in the U.S., with demand growing about 15 percent annually. Factoring in an annual loss of perhaps 10,000 programmers who retire (a figure sure to increase after the year 2000), you find there are about 310,000 programming jobs open this year. By the year 2000, that figure should rise to about 415,000 positions available.
To meet this demand, U.S. universities will turn out about 46,000 computer-science graduates and 31,000 electrical engineering graduates this year; some 8000 of these graduates are foreign students who will return home. In addition, there are approximately 20,000 MIS graduates and perhaps 30,000 permanent immigrants who will enter the U.S. programmer market. All in all, this translates into about 119,000 new programmers for the 310,000 open positions. By the year 2000, this figure should increase to about 184,000 new people, although, again, the demand will be for 415,000 programmers.
To a great extent, the problem of programmer supply and demand is nothing new. The rise in demand has been there for years, but the deficit has been masked by massive downsizing in MIS, aerospace/defense, and large computer firms, not to mention the simultaneous growth of software service exports from India. Software publishers have been the last to feel the pinch.