Issue No. 03 - May/June (1999 vol. 16)
Most of you have heard about the skills shortage in the software industry—you may even be tired of hearing about it. In this issue, we present three articles that address several interesting aspects of this topic, from three completely different perspectives. As usual, Capers Jones offers hard numbers shedding light on the extent of the problem. Subroto Bagchi gives us a fascinating perspective about efforts in India to help the rest of the world with this problem while elevating the Indian export business to breathtaking levels. Finally, Ahmed Seffah looks at a private-industry attempt to fill a gap in the traditional educational system.
First of all, let's put aside any doubts as to whether this problem is real or imaginary. Some skeptics claim that the reported shortage of about 200,000 software workers in North America (including a shortage of 20,000 in Canada) is a ploy by special-interest groups to open the immigration tap to an inflow of low-wage workers who would take work away from local talent. There are several arguments against such a hypothesis. I was a member of the US advisory panel for a study initiated by the US Department of Commerce to investigate the labor shortage in 1998. The panel was chaired by Howard Rubin and included such well-known experts as Capers Jones, Lawrence H. Putnam, and Ed Yourdon.
JUST WHAT IS THE PROBLEM?
The conclusions were clear and irrefutable: there is a severe shortage in skilled software workers in the US. 1 It currently amounts to about 200,000 and it will continue to rise at an alarming rate. A separate study that I chaired in Canada came to the same conclusion: the shortage is about 10 percent of the current labor force in the software industry.
We have other, more indirect indications that support these findings. The salaries paid to programmers, analysts, and their managers are spiraling out of proportion. Compensation packages often include stock options, signing bonuses (a sure sign of fierce competition for a scarce resource), and other benefits that were not offered in the past. Salary surveys can be flawed, and their findings depend on such variables as geographic distribution and industry segment. Nonetheless, a junior programmer right out of university can earn $50,000 a year or more in Silicon Valley. An experienced senior designer can expect well over $100,000 a year apart from stock options and bonuses. If you've read the prospectuses for Initial Public Offerings of software companies, you will likely find under the risk section a comment about the potential failure to meet targets based on the inability to recruit enough software developers. The same concerns are frequently expressed in annual reports.
Finally, we should reflect on countries such as India that systematically go about building an educational infrastructure capable of producing large numbers of new software development talent, mainly targeted for export. It strikes me that these initiatives do not result from accident but rather reflect a systematic approach to filling an increasing demand.
WHAT CAN WE EXPECT IN THE FUTURE?
The question is whether we are experiencing a short-lived cycle, which inevitably will be followed by a downturn, or a situation that will continue to worsen. I believe it will get a lot worse until the economy adjusts or until we finally find the elusive "silver bullet." There are several reasons.
- First, the current excess demands for Year 2000 and Euro fixes as described in Capers Jones' article will not completely disappear on 1 January 2000. Many of these fixes are temporary patches, and once we make it through the turn of the century, many companies will have to go back and rework their software with more permanent solutions.
- Second, the Internet has only just begun to create an enormous demand for a new generation of Web and Java designers. The current state of the Internet phenomenon represents only the first "baby steps" of a whole new industry. As the baby grows, so will its appetite for software.
- Finally, the world has become used to having just about every object, from cars to toasters, equipped with computers. The hardware chip density has for many years followed Moore's law of doubling in capacity every 18 months, and experts agree that there is no end in sight to this trend. The increase in hardware power will continue largely independent of advances in software technology. As hardware performance increases, so will the demand to feed this hardware with software.
My own projections (to be published in the conference proceedings of PICMET, July 1999) predict, under certain assumptions, a potential shortage of a million software workers by the year 2006. As an aside, it is interesting to note that India is planning to grow a million new programmers in roughly the same time frame.
Looking at the combined results from studies in the US and Canada, the rapidly rising cost of resources, and off-shore efforts to boost the GNP of countries like India, China, and East European countries through software skills exports, the evidence of a growing demand for software developers that outstrips at least the local supply seems irrefutable.
WHAT ARE THE IMPLICATIONS?
For many years I was able to make a surprisingly good guess at the gross revenue of software companies (companies that produce software as their core business) by multiplying their total staff by $100,000. As usual, there were exceptions, as a quick look at the Microsoft Web site shows ($650,000 per staff member). But with average annual salary and benefit costs reaching about $75,000 per software developer, and considering a 1.75 multiplier for overhead, it quickly becomes obvious that the old rule of thumb no longer works. (For those inclined to analyze numbers in great detail, I admit that this is not an economic thesis—I am just using ballpark numbers to illustrate the point.)
The result is that either productivity must increase or the cost of software will have to go up. The other alternative is that many companies may go out of business or have to merge with others to create larger companies with better efficiencies of scale and a stronger position in the market. Something will have to change because even the most gullible venture capitalist will eventually walk away from no-win situations.
The above reasoning is linear, and assumes no fundamental changes in the current system. Our industry is, of course, studded with very bright individuals and we know they will not just sit and watch. People will work on new approaches to software development. The articles in this section offer many short- and long-term strategies which, if applied together, may well solve the problem.
Many approaches, however, fall into a category I call "volume solutions." By that I mean solutions that simply counter an increasing demand for resources by equally increasing the supply of these resources. This approach requires the least creativity: simply import (physically or electronically) more programmers, retrain more people from other professions, or increase the output from educational institutions. These very likely comprise one necessary ingredient to the solution space. But in the long run, they cannot outpace sustained growth since even these new sources will be depleted eventually.
Furthermore, we should be cautious about simple extrapolations of current demand for programmers. It reminds me of similar erroneous predictions some 75 years ago: as the telephone rapidly increased in popularity, some predicted that within a few decades the worldwide demand for telephone operators would require that every working woman on earth be employed in this role. The advent of electromechanical and later electronic telephone switches rendered this prediction absurd. So far, an equivalent "silver bullet" solution for software development still eludes us.
One way out is to get smarter in our approach to building software. According to sources such as Capers Jones, at least 25 percent of all projects are cancelled in the later stages of the development life cycle (test and integration). This is an appalling number. Just imagine if the construction industry had a similar track record—how many unfinished high-rises would clutter the downtowns of our cities?
Closely related and similarly appalling is the maturity level of the majority of our industry: according to Howard Rubin's worldwide benchmark study of 1997, only 13 percent reportedly operate at SEI CMM level 3—the "defined" level of process maturity. The commonly accepted approach to developing software systems could be seen as equivalent to professional malpractice. Just imagine if we could somehow raise the average maturity level of our industry to CMM level 3 (or even level 4, the "managed" level)! It would not require a big stretch of the imagination to assume that we could cut the number of unfinished projects by half or better (not to mention reducing overruns of those projects that do not get cancelled). We would wipe out the labor shortage instantly!
How much does it have to hurt before we come to our senses and start thinking about a better way to conduct our profession?
Wolfgang Strigel is the founder and president of the Software Productivity Centre Inc. ( http://www.spc.ca), an international resource center for software development organizations. He has over 20 years of experience in software development and management. Prior to founding SPC, Strigel was a vice president at MacDonald Dettwiler, a Canadian aerospace company, where he introduced a successful software engineering process improvement program using methodologies, standards, and metrics.Strigel received a B.Sc. in computer science, an M.Sc. in computer science from McGill University, Montreal, and an MBA from SFU, Vancouver, Canada. He is a member of ACM, CIPS, and PMI, and is a senior member of IEEE. He is a member of the IEEE Software Editorial Board.