“The Real Software Crisis” [BYTE Magazine, January 1996]

[This commentary of mine, published in the January 1996 edition of BYTE Magazine (p. 218), used to be available on-line at BYTE.com, but no longer is, so I’ve re-posted it here. In the 17 years since I wrote it, I’ve seen nothing to change my mind.]

The shortage of top-notch programmers threatens to become the limiting factor in software development.

Grady Booch calls it “one of the dirty little secrets of software engineering”: Success in software development depends most upon the quality of the people involved. In his book Software Engineering Economics (Prentice-Hall, 1981), Barry Boehm rigorously shows individual and team productivity to be the leading predictor in estimating software costs; it’s twice as significant as product complexity. And any decent book on building and managing a software team urges you to “hire the best people.”

Ah, there’s the rub. Perhaps you’ve noticed that it’s getting more and more difficult to locate and then hire the best people. This isn’t an illusion; it’s real, it’s significant, and it’s only going to get worse. It is, in fact, the heart of the real software crisis: There is more software to be developed than there are capable developers to do it. Demand will continue to outstrip supply for the foreseeable future. Hence, more and more software will be behind schedule, over budget, underpowered, and of poor quality — and there’s nothing we can do about it.

“Nothing?” you ask. “What about better schools, better tools, better on-the-job training, better methodologies?” Those are essential because they raise the quality of most developers. But they do not solve the fundamental problem.

The conclusion I have reluctantly come to after more than 20 [and now nearly 40] years of software development is this: Excellent developers, like excellent musicians and artists, are born, not made. The number of such developers is a fixed (and tiny) percentage of the population. Thus, the absolute number of such developers grows very s lowly. At the same time, the demand for them expands rapidly due to the world’s increasing use of, and reliance on, software.

The situation is worse than it appears. Some of these innately talented people never go into the computer industry. Many who do never develop their full potential. Others become prima donnas, demanding large salaries and extreme benefits. Or they become “cowboy programmers,” shooting from the hip and holding teams, projects, or entire companies hostage. A few burn out and leave the field. Of those left, only a fraction meets the requirements for your project.

This is not to slight the decent, talented software engineers, the ones who study hard and work hard at developing and maintaining their skills. Indeed, if not for them, we wouldn’t have a software industry at all. But even they can’t meet the demand, and their efforts are undermined by the mediocre (or worse) programmers.

Suppose that companies needed in-house orchestras to produce quality music. The demand for musicians would explode, and their starting salaries would soar. Think about the vast number of people (especially college students) who would take up music regardless of their talent or inclination, guaranteed an entry-level position because companies were desperate to fill all those seats.

Here and there, a great soloist or ensemble would play with passion, skill, and creativity, but much of what musicians produced would be cacophony, especially as the newcomers and those lacking talent and skill struggled with unfamiliar instruments and pieces. Now imagine that jobs, companies, and lives depended on how flawlessly the players performed.

This scenario describes the current state of software development. I saw this for the first time when I went back to teach computer science at the same place I had studied a decade earlier. The enrollment of computer science majors had gone up by almost a factor of 10, yet a professor who had been there since I was a student noted that the absolute number of really excellent computer science students remained the same.

In the years since my graduation, my experience in building and working with development teams has confirmed this. Ask any manager how many resumes and interviews it takes to find a top-notch software architect or developer.

As needs, costs, and opportunities escalate, some great breakthrough — a “silver bullet,” to use Frederick Brooks’ classic phrase — may yet quell the software crisis, reducing the complexities of development. If you wish to bet on that, go ahead. But in the meantime, keep your eyes and your checkbook open and grab the great programmers when you can — they’re getting really hard to find.

[And, no, I have seen no silver bullets to date. Note that this recent (June 2013) article reaches the same conclusion: the real shortage is a shortage of talent. In the meantime, I’ve expanded my qualifications of an outstanding software engineer, but talent remains the starting point.]

About the Author: bfwebster

Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com.