Over the last forty years, a small set of classic works on risks and pitfalls in software engineering and IT project management have been published and remained in print. The authors are well known, or should be: Gerry Weinberg, Fred Brooks, Ed Yourdon, Capers Jones, Stephen Flowers, Robert Glass, Tom DeMarco, Tim Lister, Steve McConnell, Steve Maguire, and so on. These books all focus largely on projects where actual software development is going on. A new book by Phil Simon, Why New Systems Fail, is likewise a risks-and-pitfalls book, but Simon covers largely uncharted territory for the genre: selection and implementation of enterprise-level, customizable, off-the-shelf (COTS) software packages, such as accounting systems, human resource systems, and enterprise resource planning (ERP) software. As such, Simon’s book is not only useful, it is important.

Go read the whole thing. ..bruce..

I was reading the comment thread to this Slashdot post on the declining percentage of women studying computer science. All the explanations you would expect are offered, with a fair amount of point and counterpoint. But one commenter offered a very real example of the difference between computer science and the various professional engineering fields (civil, mechanical, etc.):

My wife and I have been married for 31 years. We met in college. She was a civil engineering major, I was a computer science major. She later changed her major to mechanical engineering when she learned that ME’s are more widely employable than CEs. When we met she was a freshman and I was a senior.

I went on to get a masters degree, she took the classes for a master degree but spent the time she would have spent on a thesis getting ready for, and passing, the P.E. exam. She has had her stamp for a long time.

We are both now in out fifties. She gets calls several times a year offering her jobs. Some in the private sector, some in the public sector. People value her decades of experience. People look up to MEs with decades of experience and a professional certification.

I was laid off for the last time on my 49th birthday and have not been able to find a technical job since. It is hard to find a company that will believe that I actually have the experience I have. I can’t tell you how many times I have had an interview where I have been challenged on my experience and even though I can prove every bit of it people just don’t believe it. And, don’t get me started on certification for computer people, compared to getting a PE certification in the computer world isn’t even a bad joke. It is mostly just a con.

Both sexism and ageism are quite rampant in the IT field. Much of the hiring that goes on consists of bringing in cheaper labor willing to work long hours without overtime. Such an approach is, I believe, counterproductive, but that’s true of much of what passes for IT decision making and management within corporations and government agencies.

The problem is that no certification and licensing standards were set up 30-40 years ago, when they should have been. The debate raged — we discussed it at length in my CS 404 class as an undergrad — but the problem was that software engineering was just then being created, and there was no consensus on what the governing standards and practices should be.

Thoughts?  ..bruce..

After years of fretting over coming shortages, the country is actually facing a dwindling number of young people entering engineering and technology-related fields.

Universities call it “rikei banare,” or “flight from science.” The decline is growing so drastic that industry has begun advertising campaigns intended to make engineering look sexy and cool, and companies are slowly starting to import foreign workers, or sending jobs to where the engineers are, in Vietnam and India.

It was engineering prowess that lifted this nation from postwar defeat to economic superpower. But according to educators, executives and young Japanese themselves, the young here are behaving more like Americans: choosing better-paying fields like finance and medicine, or more purely creative careers, like the arts, rather than following their salaryman fathers into the unglamorous world of manufacturing.

The problem did not catch Japan by surprise. The first signs of declining interest among the young in science and engineering appeared almost two decades ago, after Japan reached first-world living standards, and in recent years there has been a steady decline in the number of science and engineering students. But only now are Japanese companies starting to feel the real pinch.

Read the whole article.  ..bruce w..

So, why would IT engineers leave one of the most highly regarded, high-quality, and successful IT organizations on the planet, viz., Google? This collection of ‘farewell’ notes from departing Google employees gives some clues (hat tip to Valleywag):

For the last two years, I have had a fantastic time helping to build Google Webmaster Central. I have loved working with the (ever-expanding!) team, writing about search on the blog and for the help center, and designing features for the webmaster community. (…) Now I have an all-new opportunity to work on the unique challenges of the vertical and local search space at Zillow. (…) Making the move was a very difficult decision, but the challenge of creating something new in a space that’s so young and evolving was too great to pass up.” (Vanessa Fox, Google Webmaster Central Product Manager – June 14, 2007)

“Today’s my last day as an employee of Google. I’ve been on leave since December, so it’s not really a big change this day. But now the decision’s made. It feels a bit strange leaving such a great and productive company. But I’m ready to do something new with a smaller group of people.” (Nelson Minar, he created Google’s first APIs – April 7, 2006)

Valleywag summed it up as “to be in charge”, but the postings also make it clear that in many cases it has to do with the size of the organizations. Setting aside my own firm, I’ve worked for organizations ranging in size from 2 employees to over 150,000 employees (PricewaterhouseCoopers in the 1999-2001 timeframe); I still lean towards the smaller size. I suspect in many cases there is a third reason as well: a shot at more valuable stock options, particularly for those who joined Google relatively late.

On the other hand, The Secret Diary of Steve Jobs has a somewhat harsher (though not necessarily incompatible) take on it about a month ago:

You’ve got these weirdly smart and semi-nasty super-spoiled children who really believe they’re superior beings who shouldn’t have to work too hard and who really don’t take criticism well (because they’ve never received any in their sheltered little lives, and it just totally knocks them on their ass) and on top of all that they are almost entirely incapable of focusing on anything for more than a few minutes at a time. You’ve got an entire corporate culture built on ADHD and entitlement. Nice work, frigtard.

Plus you make a big deal of only hiring these super-high-IQ kiddies and the fact is that most of them truly are smart, but then you put them into this horribly dull and easy drone work on AdWords and AdSense and they’re all bored to tears and totally disappointed because they really really really thought they were going to do something meaningful with their lives and now they’re just worker bees — pampered worker bees, sure, but still — and maybe they should have taken that offer from McKinsey but they really thought Google was going to be so cool and blah blah blah.

So, maybe it’s the Dead Sea Effect after all — just on a much higher level. ..bruce..

Alex Papadimoulis over at The Daily WTF (one of my favorite IT blogs) has posted a lengthy and thoughtful solution to the problems I raised in my post on the “Dead Sea effect“. Specifically, he refers to the “Up or Out” model, pioneered over a century ago by Paul Cravath and well-known to anyone who has worked in a law firm or one of the Big Eight Six Five Four (how many are left now?) Consulting Firms.

In fact, I’m quite familiar with it, because I spent two years as a Director (one level below Partner) at PricewaterhouseCoopers, which definitely used the “Up or Out” model. In addition, I’ve been serving off and on as a expert witness in IT-related litigation for nearly a decade and so have spent many long hours in law firms and with lawyers (ranging from associates to partners), and as Alex notes, law firms tend to use this same model as well.

So, how well does this model work in developing and maintaining top talent? Well, that depends upon what you define as “talent”. I haven’t read Cravath’s book, so I don’t know what he originally proposed. But in modern law firms and large consulting firms, “talent” is mostly defined as “bringing in clients with lots of money” and “keeping the staff under you fully occupied with billable hours” (though, to be fair, it also includes your actual performance over the course of multiple engagements). Let me explain.

Modern “up or out” firms typically have four to six basic job titles, sometimes with subdivisions. (For example, in our division at PwC, we had — as I recall — Partners, Directors, Senior Managers, Managers, Senior Associates, and Associates.) Ideal engagements follow the “pyramid model” — say, a Partner, one or two Directors, two to four Managers, and four to eight Associates. (You scale up all the numbers for larger engagements, and so on.) In a firm like this, everyone below Partner is typically on a fixed salary, but has a billable rate for clients that was roughly some multiple of their salary. So, for example, in such a firm an associate might be paid $70,000/year but might be billed out to clients at $140/hour (equivalent of $280,000/year, assuming 2000 hours/year in billable time).

The two key words are “leverage” and “utilization”. You want an engagement with leverage — that is, one in which you could apply the pyramid model, staffing it primarily with Managers and Associates. You also want an engagement that would allow high utilization — that is, it would keep those Associates (and hopefully the Managers) billing 30+ hours/week (80% utilization) throughout the course of the engagement.

The “Up or Out” process itself can be pretty brutal. Here’s a simplified example, again typical of such firms. Once a year, each level would do stack rankings on a fixed curve of all those on the level beneath them. So, for example, the Managers would have to stack-rank all the Associates — if you had 20 Associates, you would have to rate them #1 through #20, with no ties. Then you would have to apply a fixed curve — say, 10% A, 20% B, 40% C, 20% D, and 10% F. That would mean that of those 20 Associates, only #s 1 & 2 could be A-rated, #s 3-6 would be B-rated, and so on, regardless of how much or little separated them. Those not making the minimum grade (whatever that was) would be invited to leave. Even those who were above the cut but who did not score high enough for a certain number of years running would be informed that there was no promotion to Manager in their future and that they should start looking elsewhere. The Directors would repeat the same process for the Managers, and the Partners for the Directors.

(All that said, let me add that the mentoring aspect that Alex mentions was very real and extremely valuable. I learned most of what I know about being an expert witness from PwC — specifically from Jeff Parmet, the PwC Partner who hired and supervised me, as well as from other PwC Directors and Partners, not to mention the significant in-house training sessions that PwC ran each year. I enjoyed my two years at PwC and am grateful for them, particularly when I find myself up against an expert witness who has not had such a background.)

Now, if we look at applying “Up or Out” to an in-house IT department, two immediate questions arise. First, what do we define as “Up”? Most large organizations really don’t have a career track for IT engineers above “senior programmer”, except for maybe a handful of “architect” slots and possibly a Chief Technical Officer (CTO) position. The usual “promotion” is to move onto a management track, which a lot of IT engineers don’t really want and aren’t necessarily very good at. This is, I believe, one of the factors behind the Dead Sea effect — your talented IT engineers see very few choices ahead within the organization that let them keep doing what they do best, so they leave; the less talented/skilled IT engineers, on the other hand, are content to remain in their current positions and not advance at all.

I have proposed before to organizations that they implement a non-management technical track all the way to the CxO level, because it’s a lot cheaper to keep your best people on salary than to hire them (or their equivalents) back as consultants at 2x to 6x their salaries. Such a track might look like this: Associate Engineer -> Engineer -> Senior Engineer -> Technical Officer -> Senior Technical Officer -> Executive Technical Officer -> Chief Technical Officer (only one of these). The ranks from Technical Officer and up would have salaries, perks, and benefits equivalent to progressing through management (VP, Senior VP, Executive VP), but without actual management/head count responsibilities. Architects, mentors, and project overseers would be drawn from these ranks. I honestly believe that this upper-level technical track would save most organizations anywhere from millions to hundreds of millions of dollars in failed or late IT projects because they would constantly disrupt the ‘thermocline of truth‘ before it even formed.

The second question is, how, and on what basis, do we evaluate the IT engineers? This is a trickier question than it would appear at first. That’s because whatever criteria you select for evaluation will then become the key factors that the IT engineers will “game” to. Suppose, for an example, that you judge IT engineers on the basis of accurate schedule prediction and on-time completion of subprojects. You will suddenly have a group of hyper-conservative IT engineers who maximize schedule estimates and minimize completion criteria in order to ensure that they have a great (for them) track record. There’s actually some upside to that — you’ll dampen the inherent (and often excessive) optimism found in many IT engineers — but you have to be willing to live with the consequences as well (such as very long, drawn-out projects).

As for the ‘how’ question, I think that a group evaluation from two levels up might work the best; that is, have all the Senior Engineers evaluate the Associate Engineers, the Technical Officers evaluate the Engineers, and so on. I think the extra distance will allow for a more objective evaluation; IT engineers are notorious for being competitive. I don’t even mind stack ranking so long as there is no fixed curve. If you’re doing your recruiting and hiring correctly, you shouldn’t have many (if any) D- or F-level IT engineers.

So, Alex’s proposal has definite merit, but has some potential pitfalls as well. I’m curious to see what more details he may suggest for it. ..bruce..

[Copyright 2008 by Bruce F. Webster. All rights reserved. Adapted from Surviving Complexity (forthcoming).]

As new technologies become hot markets, the number of skilled developers is small, and they command high salaries. For example, in the software industry this has been true at various times for 8086 assembly, Windows, C++, OLE 2.0, object-oriented development, Java, .NET, Python, and subsequent technologies. Each area fills in with time as sustained demand and high salaries draw more engineers into it. But, curiously enough, the absolute number of excellent developers in a given area of technology remains pretty much the same.

When I was teaching computer science at Brigham Young University in 1985-87, the number of students enrolled as computer science majors had increased dramatically — by a factor of five or so — from when I had been a student there a decade earlier. One of the professors, who had been around since the early 70’s, observed to me that the number of really good students in the department was still pretty much the same; the five-fold growth of enrollment hadn’t brought a five fold, or even a two-fold, increase in excellent CS majors. Why? Because those students with interest, aptitude, and native talent has been signing up all along; the surge in enrollment had come from students who saw computers as a way to get a great paying job, much as my friends during my undergraduate days had signed up for pre-law or pre-med.

When a new, powerful technology emerges, the best and the brightest are the ones who rush in there first. They are the ‘early adopters’ and they quickly master the technology. Other IT engineers follow suit as acceptance of the technology grows — but many of them will not be as talented or capable. The number of truly excellent developers for that technology increases quickly, then increases slowly, then finally levels off, achieving something of a steady state, with some excellent developers moving on to newer technologies as other excellent developers come to that technology for the first time. Think of a pool of water with several streams flowing in and several more streams flowing out, keeping the pool level itself steady.

Because of this, I’d like to propose a minor addition to the vast assortment of laws and rules governing technology and engineering:

• The Expert Pool: the number of excellent developers in a new area of information technology quickly reaches a constant value, which is sustained through the period during which the technology is vital, then slowly declines as the technology becomes less relevant.

This is actually a corollary to the apparent fact that there is more or less a fixed number of really excellent developers in IT.

Note that it isn’t always the same excellent developers who move on each time. There are always new developers (excellent or otherwise) entering IT and old ones leaving (or dying). Many excellent developers remain with a given technology due to job or career demands (or choices); they stay in the pool, so to speak. Some developers become expert in that technology by virtue of the amount of time they spend in the pool.

The upshot, however, is that most organizations seeking outstanding talent in a given technology are going to be competing for the same limited pool of experts. Anyone who had done extensive recruiting for a given technology knows the phenomenon of seeing the same names show up again and again. Often these experts end up as consultants, authors, and/or conference speakers and may thus take themselves out of consideration as a full-time employee.

Two disappointed believers,
Two people playing the game.
Negotiations and love songs
Are often mistaken for one and the same.

– “Train in the Distance”, Paul Simon

I used to have arguments with Carol Teasley, one of my mentors, regarding software development methodologies. She contended that there were no really good or effective methodologies, because if there were, everyone would use them and they would work. I, in turn, contended that there were several effective methodologies, but that they were just badly applied for the most part.

Then while watching the movie “A Beautiful Mind” (about Nobel Laureate John Forbes Nash, a brilliant mathematician who struggled with schizophrenia), I was struck by the sequence that represented (not entirely accurately) some of his insights about game theory and multi-player equilibrium points. It occurred to me that software development within a typical corporate/government organization is really an instance of multi-player game theory, with several general classes of players. The real issue in organizational software development is not the development methodology being used. That methodology, at best, is necessary but not sufficient for a successful project; it may actually be irrelevant, and at worst it may be a hindrance.

The real issue is the game that’s being played.

Organizational software development, I believe, can be thought of as an n-player game, where each set of players has voiced and unvoiced (and possibly even unconscious) goals, many of which are incompatible with those of one or more other players.

At the most abstract level, the fundamental division is among IT developers (Geeks), upper management (Suits), and end-users (Users). Put very roughly, the Geeks work as if they’re playing Halo, the Suits work as if they’re playing poker, and the Users are just trying to balance their checkbooks. The result is that the three groups mostly baffle and frustrate one another; they can’t even agree on what game they’re playing, much less what constitutes winning.

Adding to the problem is that each group has generally negative perceptions of the other two, which in turn shape the games. For example, Geeks tend see Suits as “marketing weasels” who make impossible promises (to Users) or demands (of the Geeks) that the Geeks can’t possibly meet. Geeks look upon Users with a mixture of condescension and dismissal, though that elevates to horror and dismay if the Geeks have to directly provide tech support to the Users.

Suits tend to see Geeks as overgrown, naive, but possibly bright children who never actually deliver anything. They tend to see Users as annoying sheep, except when the Users control the purse-strings, at which point the most far-fetched User demand is seen (and portrayed by the Suit) as eminently possible.

The Users tend to see Geeks as irascible wizards who may or may not deliver the magic necessary to make their own lives easier. They tend to see Suits as corrupt nobles who hold not just power but also the wizards’ collective leashes.

Now, these are of course blatant and gross oversimplifications, but ones with a foundation in reality; were it not so, the Dilbert comic strip would have died a quiet death long since.

My next set of posts will look a bit closer at these abstract classes (Geek, Suit, User) and then will look at some of the real-world patterns (internal development, external development, and so on).

[Copyright (c) 2006 by Ruby Raley and Bruce F. Webster. All rights reserved.]

Managing talent and skilled workers is a key success factor in the 21st century across all industries, not just technology-based industries. However, many of our organizational practices originated in the 1800s with railroading and the rise of the Industrial Age. Your average Gen Y programmer bears little resemblance to the unskilled and semiskilled laborers of the past. Just as technology has evolved, so must our organizational practices.

We have observed that firms (corporations, government agencies, etc.) with consistently successful IT organizations share certain traits with sports teams — specifically, American football teams. This article looks at what makes a successful football team and then applies those traits to IT organizations. [Continues after the jump]

Personnel

Software engineering studies repeatedly point out that the people involved represent the single most significant success factor in an IT project [2, 3, 8], yet most firms treat IT personnel as though they were in a “parts from the lowest bidder” situation. Many firms struggle with their IT organization because of lack of attention to personnel issues — in particular, recruiting IT personnel, assigning them to appropriate positions, and coaching them for best performance. Most firms do not ensure strong relationships and goal alignment among IT and business stakeholders.

Recruiting

Numerous studies in software engineering have shown wide variations in productivity and talent among IT personnel, even when education and experience are equivalent [8, 11]. Other studies have indicated that smaller IT teams tend to do better than larger IT teams, due to issues such as communication [4]. Still other writings point out the strong need for teamwork and cooperation within IT development groups [6,7].

Unfortunately, most firms do not recruit IT personnel with those factors in mind. Instead, they focus on checkbox items, such as certifications and years of (claimed) experience with a given technology. They seek to match keywords against what they think their needs are. They conduct at most a few interviews with a given candidate — typically by managers, with at most one technical interview. They often fail to evaluate candidates by talent, skill, attitude, and ability to cooperate — by and large, they do not know how. And they tend to believe, unconsciously or explicitly, in the mythical man-month, thinking that the larger their IT staff, the quicker they can complete projects [5].

By contrast, sports teams invest significant time and resources in recruiting a small number of players. They read the sport press, review scouting reports, watch game films, analyze performance statistics, conduct interviews and physical examinations, and may require a tryout before agreeing to take on a given player. They may have a probation period during which the player has to either make the team or be let go. They recognize that they have a limited number of slots on the team, and they may also be operating under financial constraints, either due to the team’s own resources or a league-imposed salary cap. They then negotiate a contract commensurate with the player’s talent and skills, physical condition, and performance to date, as well as their own need to fill a given position, any financial restrictions they may face, and competing offers from other teams.

How would this look in an IT organization? Here are some ideas:

• Recruit based on reputation and recommendation. We have found that the single best predictor of performance of an IT engineer is a strong recommendation from an IT engineer we already trust and respect. If you don’t have an existing recommendation, then have all your best IT people interview the candidate.
• Focus on talent and professionalism. Talent is critical, often more so than specific technology experience. A talented IT engineer can pick up new technologies rapidly, but an untalented IT engineer, however experienced, will be a drag on the project. At the same time, having talent is no excuse for being a prima donna. We’d rather have someone who is a bit less talented but gets the work done. Unfortunately, we lack strong metrics for predicting talent. This is why the recruiting process is so important, as we discuss below.
• Work with a team size limit and a salary cap. It is our observation that firms often have too many IT engineers rather than too few. This is often a direct result of management policies and politics that equate headcount with internal clout — in short, kingdom building. That can be a hard hurdle to overcome, particularly since such politics are often entrenched in the firm.

At Pages Software (a venture-funded startup where Bruce was CTO), we applied these techniques to build our engineering team from scratch. All Pages engineers made recommendations as to which candidates to bring in for interviews, either from incoming résumés or from their own professional network. When a candidate was selected to come in for interviews, every Pages engineer would conduct his or her own one-on-one interview with that candidate, each of us using our own preferred style. We would then meet to evaluate how that person would meet our current and future needs and how well that person would fit into our team. We only had a certain number of slots, and we were going to be working very long hours together for years, so we chose carefully. The result was a very stable and talented development team that had almost no turnover over a grueling four-year period.

Recruiting is extremely important, so staff the function carefully and be sure to work closely with your new team members so that they can become productive quickly.

Organizing the Team

Again, many studies have recommended ways IT teams should be organized, starting as far back as 1969 [1] up to the latest debates surrounding agile development [10]. However, in focusing on organization, we often lose sight of what it takes to build a team.

A Quarterback

The quarterback is the most visible player on a football team and is responsible for leading the offensive team to get the ball into the end zone. The equivalent position on a software team is the chief architect. Fred Brooks first laid out the need for a chief architect at length, most particularly for conceptual unity [5]. We believe that his observation stands but that it is too often ignored. For example, one project we consulted on had 30 engineers developing network management software for a global telecommunications project — but no architect (or architecture) at all! Diagrams existed for half a dozen subsystems, but those diagrams did not connect up with each other. One of our first tasks was to create an architecture for the project, then adjust the subsystems to fit within that architecture.

Mutually Supporting Team Members with a Common Goal

This may sound obvious, but think: how many of you can say this describes your IT department? Imagine that our hypothetical football team has just scored, and the receiver who caught the ball is doing a victory dance for the camera and the local fans. Does the culture at your firm reward “displays of unsportsmanlike conduct” or penalize that behavior? There are IT shops where a flourishing hero culture exists. Management rewards those individuals almost exclusively, while their fellow team members smart from their jibes and never get to carry the ball.

We remember a release cycle at one firm that illustrates how teams can be torn apart. The senior architect delivered a new release of middleware to the product team with no documentation and no test scenarios. A junior QA engineer started the validation of the middleware, working day and night to test and document the system. After days of work and unanswered pleas for help, he sent his team lead and the architect an e-mail stating, “It doesn’t work,” and went home for the night. The architect’s response? He sent a note to the entire team, including the CTO, accusing the junior team member of leaving the ball on the field and insinuating that his incompetence was the real issue. The result? The release never delivered a performance improvement, team communications suffered for months, yet the architect was rewarded and touted by management. [UPDATE: Here's a great real-world example of nearly identical behavior.]

Successful IT organizations need clear technical leadership at multiple levels that is supportive and makes strong contributions to team success.

Coaching

Once you have what you consider to be the right personnel in place, you still need to coach them. Professional sports teams recruit the best of the best, pay them millions of dollars, and yet still focus significant amounts of time and money on guiding these top-notch athletes to superior performance through coaching and the use of playbooks. Why is it that we think we can hire IT professionals off the street, throw them into cubicles, and then expect them to work as a team to achieve the firm’s business goals via technology?

Coaching Staff

A professional American football team typically has a head coach with several specialty coaches (QA, backfield, receivers, offensive line, defensive line, etc.). The head coach builds and oversees the entire team, interacts with the business side of the firm, devises how the team is going to win the championship, makes position assignments, adds or cuts players, and motivates, encourages, and/or excoriates players as appropriate. The specialty coaches work with team members in their specific responsibilities to develop requisite skills.

Each IT department and project head needs to see him- or herself as a coach. A coach is held accountable for the overall success of the team and is focused on improving both individual and team performance. Likewise, if you’re in IT management, you should focus first on building the right team for the project at hand, then coaching each IT engineer in his or her responsibility. If an individual isn’t performing and doesn’t improve under direct coaching, then find someone who will perform. Frankly, this can be tough; some very talented IT engineers, like some very talented athletes, can become prima donnas, and one of the hardest decisions you may face is telling such a person to either work as part of the team or leave it (and possibly the firm).

“Specialty coaches” would correspond to mentors, an idea much touted in management circles but seldom applied to the IT organization. At OSG Incorporated, we have seen the effectiveness of placing a handful of seasoned technical mentors — specialized in areas such as analysis, architecture, design, implementation, and quality, as well as key technologies — within large IT organizations. Mentors work with the IT engineers to solve problems, improve performance, and generally convey the hard-won experience and insight that only come from years in the trenches.

Philosophy and Playbooks

Successful head coaches are usually known for having a definite philosophy, which is reflected in the assistant coaches they hire, the players they recruit, the offenses and defenses they implement, and their approaches to player coaching and discipline, team practice, play-calling, and game strategy. Similarly, each football team has its own playbook, which contains both offensive and defensive sets that reflect the head coach’s philosophy and/or the expertise of the assistant coaches, as well as the particular strengths and weaknesses of the players themselves. The team practices these plays and sets until they become second nature; the plays and sets are then called as appropriate by the coaches, quarterbacks, and/or defensive team captains.

It’s important to note that successful head coaches don’t panic or throw out the playbook when things go wrong. They may make a few substitutions or change the offensive play/defensive set mix, but they tend to stick with their philosophy and keep doing those things that made them successful in the first place.

While much as been written, starting largely with Brooks [5], about the need for conceptual unity of system architecture, less has been written, said or done about conceptual unity across the entire IT department — the equivalent of an IT coaching philosophy. In our experience, few IT organizations apply a consistent IT philosophy to such diverse issues as software methodologies, enterprise-wide technical architectures, software architecture and design patterns, coding standards, development tools, reusable deliverables, QA processes and tools, and component- and service-oriented architectures (not to mention recruiting, training, and assigning IT personnel).

As a result, we have observed significant mismatches, conflicts, and gaps among those aspects of specific IT organizations. For example, we have consulted at a firm that sought to follow a highly structured formal methodology while at the same time implementing a “weekly software build” schedule. The result was that, strictly speaking, developers only had about eight hours to do coding each week, since the rest of their time was taken up by paperwork and formal design, implementation, and testing reviews.

By contrast, we have worked with senior IT executives who had a set of explicit rules and principles that they typed up, handed out, and would cite when decisions and issues came up regarding the various processes, tasks, and technologies cited above. Indeed we have done this ourselves on occasion — we have found this prevents many problems and simplifies many others. A good template is the Agile Manifesto and the associated “Principles behind the Agile Manifesto” [10]. We say this not because we unconditionally endorse the agile approach, but because the Agile Manifesto and Principles together form an outstanding example of a self-consistent IT “coaching philosophy,” all expressed in fewer than 20 statements. Another source that we both have used professionally is the list of 180 or so “heuristics for systems-level architecting” collected by Mark Maier and Eberhardt Rechtin [9].

The IT equivalent of a playbook would start with this written philosophy, explaining the principles. It would then build upon the processes, tasks, and technologies listed above, explaining how (within this particular IT organization) you work with your teammates to carry out specific key IT development tasks or solve particular problems. Even we aren’t sure just what such a playbook would look like — other than it would have to be relatively small, simple to look through and read, and easy to update.

Performance

Having recruited the right personnel and taught them the plays, the coach now has to get the team to go out and perform — do the blocking, passing, running, and tackling necessary to win the game.

We picked American football as our major analogy for several reasons; most important is the relative balance between offense and defense, which we correlate to development and quality assurance, respectively. (The other software development lifecycle activities can, for now, be relegated to either “coaching” or “special teams.”)

Offense = Development

The development team is essential to winning the game: delivering working code that meets requirements on time. And just as different football teams use different offensive philosophies, your IT organization needs to decide what offense each IT team will use.

West Coast Offense (Adaptive)

A passing offense inherently takes a greater risk in each play. There is the opportunity for interceptions and missed passes, but there is also the opportunity for fast scores and game-changing plays. The IT equivalent is an “adaptive” IT methodology, such as the various agile approaches. This makes sense when business drivers require rapid development and close interaction with end users, such as when business drivers change on a rapid basis (due to competition) and a fast, tight development cycle is required. For example, OSG has developed a technology for data extraction and visualization that can be implemented, deployed, and customized in a matter of weeks in large organizations with established IT infrastructures. We use an agile methodology because the payoff for clients comes from a very rapid turnaround time, both in getting the data now and in implementing custom visualizations in short order.

East Coast Offense (Predictive)

A running offense seeks to control the ball, grind away at the opposing team, and occasionally break loose for long yardage by focusing the best running back on the weak areas of the defense. The IT equivalent is a “predictive” or formal methodology, such as a classic or modified waterfall approach. This makes sense when business drivers and technological considerations require a significant investment in analysis, architecture, and design up front; for example, in industries where reliability is an issue and the system lifecycle may be five to 10 years. We have seen the successful application of such an approach in custom development of an enterprise-wide IT system for a utility firm, with formal signoff not just of deliverables but of the acceptance criteria for those deliverables.

Balanced Offense (Iterative)

A balanced offense alternates evenly between passing and running, picking the best play as the situation demands. The IT equivalent is a modern iterative methodology, such as the Rational Unified Process (RUP). This makes sense when you need to make some up-front investment in analysis and architecture due to the scale and complexity of the systems under development, but you don’t want to wait months or years for initial deployment; for example, when re-engineering and replacing legacy systems.

Just as in football, there are fierce proponents and critics of each of the IT approaches above. The key is to apply them on a case-by-case, team-by-team basis — and to be willing to change or adapt if the selected approach isn’t working well.

Defense = Quality Assurance

Football teams that only excel at offense usually play exciting games and often win individual games, but they seldom end up as champions. That role is reserved for teams that excel at defense as well. The same is true for IT organizations. They may have great developers and flashy deliveries, but without a solid quality assurance (QA) team, they seldom ship a completed system on time, if at all. In our experience, most IT organizations are understaffed in QA. The QA department is viewed as a checkbox in the software lifecycle and is given little recognition, management attention, or budget. Many IT organizations would likely improve their IT project success rate by doubling their QA staff and halving their development staff.

Quality assurance is more than testing, although most organizations tend to equate the two. Instead, QA include:

• Ensuring appropriate expertise (subject matter, technological, methodological)
• Publishing guidelines and standards for every aspect of the software development lifecycle
• Gathering and using metrics
• Conducting reviews
• Carrying out a full spectrum of tests
• Managing defects and deliverables
• Having a formal software release process
• Collecting feedback for future improvements

The key word for all these activities is “appropriate” — appropriate expertise, appropriate guidelines, appropriate metrics, and so on. And while the “offense” has the task of delivering working code, the “defense” — the QA staff — is responsible for ensuring the code achieve acceptable reliability, performance, and functionality before it goes into production.

Once again, we can draw upon a few general football defense concepts for our IT equivalents:

Blitz Defense (Adaptive)

In this defense, most defensive players rush into the backfield to tackle the quarterback (or running back). This reflects agile concepts such as test-before-code and pair programming [10], as well as more general concepts such as unit, class, and module testing. Tests are defined before the code is written and are implemented at the lowest code unit levels (procedure, class, etc.). Programmers work in pairs to provide immediate feedback on design and implementation decisions as the code and corresponding tests are written. Likewise, this defense encompasses such QA concepts as coding/design standards and expertise of both developers and subject matter experts. The goal is to ensure zero defects before the code is ever released to higher-level integration and testing.

Man-to-Man Defense (Adaptive/Predictive)

In this defense, most defenders have a specific back or receiver to cover; each follows his assigned player wherever he goes. This reflects integration, interface, and end-to-end testing, as well as configuration management tools and practices. The goal is to ensure that complete scenarios and uses cases can be carried out on individual applications and virtual applications (collections of applications that work together to carry out a given use case).

Zone Defense (Predictive/Adaptive)

In this defense, most defenders have a certain area (zone) on the field to defend; each covers any runner or receiver who enters that zone. This reflects category-based testing, such as performance and stress testing, computational verification, scheduling testing, and regulatory testing, along with classic defect and change control management. The goal is to ensure that individual and virtual applications meet certain system and business requirements.

Prevent Defense (Predictive)

In this defense, most defenders drop back from the line of scrimmage in order to prevent a successful long pass play. This reflects classic waterfall concepts such as delaying most testing until coding is complete; conducting whole-system tests such as compatibility/parallel testing, restart recovery testing, user acceptance testing, and production readiness testing; and performing release management (alpha, beta, production releases). The goal is to ensure that all stakeholders agree that it is both safe and desirable for the new system to go into production.

The important point is this: QA is every bit as critical to winning as development and should have equivalent resources, including personnel and preparation.

Dealing with the Front Office

Professional football teams are a business. Their purpose, beyond the ego gratification of the owners, is to make money. Usually this requires performing well enough to sell lots of tickets, merchandise, broadcast rights, sponsorship rights, and so on. That money, in turn, provides training facilities, equipment, salaries, and benefits for the players. Note that spending lots of money, even with a well-respected head coach, does not necessarily guarantee victories and championships. Witness, for example, the travails of the Washington Redskins over the past several years (although, by all accounts, the franchise itself remains very profitable).

Likewise, an IT organization depends upon the business side of the firm for its funding. The IT organization needs to “win” by meeting certain key business drivers, thus encouraging the business side to continue to make those funds available to the IT organization. Note, also, that spending lots of money on IT does not necessarily guarantee successful delivery of the intended system, much less the expected return on investment (ROI). We have personally reviewed failed IT projects costing over half a billion dollars and troubled IT projects costing well over a billion dollars.

The bad news is that, unlike in football, the business and IT sides of a firm don’t always agree on what constitutes a “victory” (even though both sides can usually agree on what a “loss” is, at least in cases of total or significant project failure). Indeed, sometimes they cannot fully agree one what the game is.

Typically, the business side has three overlapping goals for IT:

• B1 — provide required and sufficient functionality to allow the firm to operate and compete on a level playing field
• B2 — provide superior or unique functionality to allow the firm to beat its competition
• B3 — provide efficiencies in productivity to allow the firm to free up funds for investment, expansion, and/or profits

Likewise, the IT side typically has three overlapping goals for itself:

• IT1 — maintain its existing systems (and staff) or their equivalent
• IT2 — migrate off aging, obsolete, or defective technology
• IT3 — keep the business side (including end users) happy, or at least off its collective back, while getting the funds necessary to accomplish IT1 and IT2

These goals overlap and coincide to a certain extent, but there are some dangerous and often unexamined gaps in there as well. For example, say the business side proposes a project for goals B2 and/or B3. IT will see this project as a chance to meeting goal IT2 (and, with luck, IT1 and IT3) and will design accordingly. IT delivers the project into production on time and on budget, feeling that it was succeeded on all counts. However, the business side finds that the time elapsed has shifted most of the B2 (competitive) benefits over to B1 (required/sufficient), since the competition has likewise progressed. Furthermore, the increases in IT maintenance costs (due to some level of legacy system carryover) and the inefficiencies due to the inevitable disruption caused by changes to previous business processes have reduced, delayed, or eliminated the hoped-for B3 benefits.

Now that’s what can happen when IT actually does understand the business side’s requirements and delivers the system on time and on budget. Imagine the frustration and disconnect when (as is often the case) the IT and business sides cannot effectively agree upon project scope and requirements, core business drivers, or key performance indications — and the IT project is late and over budget. No wonder authors from Paul Strassman to Nicholas Carr have seriously questioned the competitive advantage of IT, in essence arguing that B1 is the only goal the business side can hope for.

We believe that B2 and B3 goals can be achieved — but that it will be harder and less rewarding unless firms rethink, perhaps radically, their entire approach to IT development and deployment along the lines we have laid out in this article.

The End Zone

We believe you will find value in thinking of your IT organization in terms of a competitive sports team and implementing some of the suggestions above. We further believe that in doing so you can improve your project success rate and increase employee retention. These concepts don’t necessarily require wholesale revamping of your IT organization; many can be implemented piecemeal or in phases, which makes them all the more attractive. The idea is to rethink your IT organization by recruiting top-notch personnel, preparing them for success, molding them into a team, and then helping them to perform in order to achieve your firm’s critical business goals.

References

[1] Aron, J. D. “The ‘Super-Programmer Project.’” Reprinted in Classics in Software Engineering, edited by Edward Nash Yourdon. Yourdon Press, 1979.

[2] Boehm, Barry W., Maria H. Penedo, E. Don Stuckle, Robert D. Williams, and Arthur B. Pyster. “A Software Development Environment for Improving Productivity.” Reprinted in Software State of the Art, edited by Tom DeMarco and Timothy Lister. Dorset House, 1990.

[3] Booch, Grady. Object Solutions. Addison-Wesley, 1996.

[4] Briand, Lionel C., Khaled El Emam, and Isabella Wieczorek. “Explaining the Cost of European Space and Military Projects.” In Proceedings of the 1999 International Conference on Software Engineering. IEEE, 1999, pp. 303-312.

[5] Brooks, Frederick P., Jr. The Mythical Man-Month. Addison-Wesley, 1975.

[6] DeMarco, Tom, and Timothy Lister. Peopleware. Dorset House, 1987.

[7] Hohmann, Luke. Journey of the Software Professional. Prentice Hall PTR, 1997.

[8] Keyes, Jessica. Software Engineering Handbook. Auerbach Publications, 2003.

[9] Maier, Mark, and Eberhardt Rechtin. The Art of Systems Architecting. 2nd ed. CRC Press, 2002.

[10] Martin, Robert C. Agile Software Development. Prentice Hall, 2003.

[11] Nash, Sarah H., and Samuel T. Redwine. “People and Organizations in Software Production: A Review of the Literature.” ACM SIGCPR Computer Personnel, Vol. 11, No. 3, January 1988, pp. 10-21.

[UPDATED (06/06/08): Ziff Davis has asked me to write a weekly "Surviving Complexity" column at their online Baseline website. My first column is here.]

[Note: some of you have asked about the Cutter IT Journal article that Ruby Raley and I wrote. It's now online here as well.]

[Copyright 2008 by Bruce F. Webster. All rights reserved. Adapted from Surviving Complexity (forthcoming).]

There are many reasons why large organizations, public and private, struggle with information technology (IT) development. One, which I’ve already discussed here and here, deals with finding and hiring the best engineers you can. But even if you do find and hire excellent IT engineers, the real question is: can you hold onto them?

There is an anti-pattern that I’ve seen in large organizations which I have come to call “the Dead Sea effect”. The Dead Sea, of course, is a large body of water between Israel and Jordan, located well below sea level. The Jordan River empties into it; water leaves only by evaporation, which means that over the eons, the Dead Sea has become very salty (e.g., 8x saltier than the ocean). As such, it is generally unable to support life, except when spring floods temporarily lower the salinity.

Many large corporate/government IT shops — and not a few small ones — work like the Dead Sea. New hires are brought in as management deems it necessary. Their qualifications (talent, education, professionalism, experience, skills — TEPES) will tend to vary quite a bit, depending upon current needs, employee departure, the personnel budget, and the general hiring ability of those doing the hiring. All things being equal, the general competency of the IT department should have roughly the same distribution as the incoming hires.

But in my experience, that’s not what happens. Instead, what happens is that the more talented and effective IT engineers are the ones most likely to leave — to evaporate, if you will. They are the ones least likely to put up with the frequent stupidities and workplace problems that plague large organizations; they are also the ones most likely to have other opportunities that they can readily move to.

What tends to remain behind is the ‘residue’ — the least talented and effective IT engineers. They tend to be grateful they have a job and make fewer demands on management; even if they find the workplace unpleasant, they are the least likely to be able to find a job elsewhere. They tend to entrench themselves, becoming maintenance experts on critical systems, assuming responsibilities that no one else wants so that the organization can’t afford to let them go.

I’m painting with pretty broad strokes here, yet I’ve seen this same effect taking place in different companies and different IT shops. Large companies tend to lose the really talented IT engineers and hold onto the less talented ones, when they should been actively seeking to do just the opposite. And the effect tends to be self-reinforcing: the worse an IT shop becomes, the harder it is to get really talented and effective IT engineers to join it and the harder it is to retain them if they do. It can reach a point that the really good talent only comes in as entry-level personnel who don’t know any better — but once they do wise up, they’re gone.

======= SOME RESPONSES TO COMMENTS =============

The discussion over at Slashdot, as well as the comments below, have raised excellent issues, though some misunderstand or mischaracterize what I’m talking about above. Here’s a response to the main themes that I see coming up there.

The Dead Sea effect isn’t unique to IT. True enough, though I could say the same thing about just about any project management issue regarding IT. What is unusual about IT (shared with other engineering disciplines) is the degree to which individual talent and other factors affect productivity and quality. And what is unique about IT (as opposed to, say, civil / mechanical / chemical engineers, architects, etc.) is that there is no standard (state-run) professional certification, so there is no assurance of minimum education and competency.

This is obvious/common sense/trivial. So are most of the problems in IT. Fred Brooks and Jerry Weinberg pretty much nailed down all the essential issues in IT project and personnel management more than 30 years ago; yet, amazingly, the problems haven’t all gone away! There is a profound lack of professional and institutional memory in IT; almost everyone who writes about IT project/personnel management (myself included) is looking for new ways to cast or explain the core issues in a touching hope that maybe this time someone will actually listen and fix them.

The Dead Sea effect is just the Peter Principle (or a corollary thereof). No, it isn’t. The Peter Principle is that a given person rises (is promoted) to her/his level of incompetence (I’m actually old enough to remember when ‘the Peter Principle’ first came out). This has nothing to do with promotion within the IT organization; it has to do with self-selected removal from that IT organization, not due to a lack of promotion or opportunity, but just because there are greener pastures elsewhere.

Not all IT shops are like this. I would certainly hope so. In fact, there are IT organizations where just the opposite occurs; the quality of the IT engineers is quite high, and engineers who are mediocre or disruptive either don’t get hired or don’t last long if they are. I worked in one such IT group (Pages Software) for five years. During that time, we had only one voluntary departure (the network admin); we had two others who were dismissed due to problems, and a few others who were (painfully) cut in downsizing. (On the other hand, here are some thoughts on why people would leave an outstanding (and lucrative) IT organization like Google.)

Not everyone ‘left behind’ is incompetent. Again, this syndrome doesn’t apply to all IT groups, and it doesn’t apply to the same extent to all IT groups. Turnover in IT personnel is common (though it can be reduced by intelligent management), and just because good engineers have left a given IT group doesn’t mean that the rest are, in fact, residue. What I’m talking about here is a very real syndrome, typically found in large corporations and government organizations, but it’s certainly not universal.

The IT hiring process is broken. Amen. Not only is the IT hiring process broken in many organizations, the entire approach to IT is often broken. It is rife with empire-building, ‘heroic’ project management, and an ‘interchangeable code monkeys’ mindset. As mentioned in the comments section below, Ruby Raley and I wrote an article for Cutter IT Journal that stated that an approach modeled after professional sport teams could well be far more effective, but no one has yet hired me to try it out. On the other hand, I would argue that this is to a large extent the approach we took as Pages, which is why we had such a great and effective IT group with so little turnover.

The problem is a failure of leadership. Again, amen. I wrote an entire book about that over a decade ago (The Art of ‘Ware, M&T Books, 1995), which I’m currently revising (see here on this website; a complete earlier draft can be downloaded for free [PDF]).

This doesn’t describe/encompass all the problems in professional IT shops. If it did, life would be much easier. Again, note that I’ve written a bit on the subject. I’m currently working on revised versions of two of my books (The Art of ‘Ware [Version 2.0] and Pitfalls of Modern Software Engineering) while writing a new one (Surviving Complexity), from which this posting on the “Dead Sea effect” is a very, very tiny extract.

Thanks for the great comments and the feedback; I wish I could get you folks to do this for all sections of all of my books.

For real-world examples of this phenomenon, just spend some time reading The Daily WTF (a site I highly recommend for all IT managers). In particular, their “Tales from the Interview” category gives some interesting insights from both sides of the hiring process.

Also, Alex Papadimoulis, who runs The Daily WTF, has put forth his own proposal for dealing with IT turnover. I especially like his concept of the Value Apex. Be sure to read his article; here’s my response to it ..bruce..

=============================================

[From The Art of ‘Ware (Version 2.0) by Bruce F. Webster (forthcoming), Chapter 2, “Supporting Development”]

In key areas of technology development, talent is scarce and salaries are high, limiting resources for other employees.

As new technologies become hot markets, the number of skilled developers is small, and they command high salaries. For example, in the software industry this has been true at various times for 8086 assembly, Windows, C++, OLE 2.0, object-oriented development, Java, .NET, Python, and subsequent technologies. Each area fills in with time as sustained demand and high salaries draw more engineers into it. But, curiously enough, the absolute number of excellent developers in a given area of technology remains pretty much the same.

When I was teaching computer science at Brigham Young University in 1985-87, the number of students enrolled as computer science majors had increased dramatically — by a factor of five or so — from when I had been a student there a decade earlier. One of the professors, who had been around since the early 70’s, observed to me that the number of really good students in the department was still pretty much the same; the five-fold growth of enrollment hadn’t brought a five fold, or even a two-fold, increase in excellent CS majors. Why? Because those students with interest, aptitude, and native talent has been signing up all along; the surge in enrollment had come from students who saw computers as a way to get a great paying job, much as my friends during my undergraduate days had signed up for pre-law or pre-med.

When a new area of technology opens up, it quickly draws to it those developers with the interest, talents and desire to become really good in it. More excellent developers do come along with time, but as they do, some of the current ones start moving on to new areas, so the absolute number stays more or less constant.

Because of this, I’d like to propose a minor addition to the vast assortment of laws and rules governing technology and engineering:

• Webster’s Constant: the number of excellent developers in a new area of technology quickly reaches a constant value, which is sustained through the period during which the technology is vital.

This may seem a bit silly or fatuous, but it’s actually critical to understand if you need to build a development team that will be working with key technologies. It’s going to be tough finding really good people, and you’ll find yourself running into the same names over and over again. The trick is getting them to come to work for you.

==========================

Compare suntzu pingfa (Chapter 2: “Doing Battle”):

A nation can be impoverished by the army when it has to supply the army at great distances. (Sonshi translation)