The Wetware Crisis: the Dead Sea effect
[Updated (09/12/13): Fixed or removed some broken links; updated some others.]
[Updated (06/16/08): Here’s a real-world project review memo, written several years ago, that described (among many other things) the Dead Sea effect.]
[Note: some of you have asked about the Cutter IT Journal article that Ruby Raley and I wrote. It’s now online here as well.]
[Copyright 2008 by Bruce F. Webster. All rights reserved. Adapted from Surviving Complexity (forthcoming).]
There are many reasons why large organizations, public and private, struggle with information technology (IT) development. One, which I’ve already discussed here and here, deals with finding and hiring the best engineers you can. But even if you do find and hire excellent IT engineers, the real question is: can you hold onto them?
There is an anti-pattern that I’ve seen in large organizations which I have come to call “the Dead Sea effect”. The Dead Sea, of course, is a large body of water between Israel and Jordan, located well below sea level. The Jordan River empties into it; water leaves only by evaporation, which means that over the eons, the Dead Sea has become very salty (e.g., 8x saltier than the ocean). As such, it is generally unable to support life, except when spring floods temporarily lower the salinity.
Many large corporate/government IT shops — and not a few small ones — work like the Dead Sea. New hires are brought in as management deems it necessary. Their qualifications (talent, education, professionalism, experience, skills — TEPES) will tend to vary quite a bit, depending upon current needs, employee departure, the personnel budget, and the general hiring ability of those doing the hiring. All things being equal, the general competency of the IT department should have roughly the same distribution as the incoming hires.
But in my experience, that’s not what happens. Instead, what happens is that the more talented and effective IT engineers are the ones most likely to leave — to evaporate, if you will. They are the ones least likely to put up with the frequent stupidities and workplace problems that plague large organizations; they are also the ones most likely to have other opportunities that they can readily move to.
What tends to remain behind is the ‘residue’ — the least talented and effective IT engineers. They tend to be grateful they have a job and make fewer demands on management; even if they find the workplace unpleasant, they are the least likely to be able to find a job elsewhere. They tend to entrench themselves, becoming maintenance experts on critical systems, assuming responsibilities that no one else wants so that the organization can’t afford to let them go.
I’m painting with pretty broad strokes here, yet I’ve seen this same effect taking place in different companies and different IT shops. Large companies tend to lose the really talented IT engineers and hold onto the less talented ones, when they should been actively seeking to do just the opposite. And the effect tends to be self-reinforcing: the worse an IT shop becomes, the harder it is to get really talented and effective IT engineers to join it and the harder it is to retain them if they do. It can reach a point that the really good talent only comes in as entry-level personnel who don’t know any better — but once they do wise up, they’re gone.
======= SOME RESPONSES TO COMMENTS =============
The discussion over at Slashdot, as well as the comments below, have raised excellent issues, though some misunderstand or mischaracterize what I’m talking about above. Here’s a response to the main themes that I see coming up there.
The Dead Sea effect isn’t unique to IT. True enough, though I could say the same thing about just about any project management issue regarding IT. What is unusual about IT (shared with other engineering disciplines) is the degree to which individual talent and other factors affect productivity and quality. And what is unique about IT (as opposed to, say, civil / mechanical / chemical engineers, architects, etc.) is that there is no standard (state-run) professional certification, so there is no assurance of minimum education and competency.
This is obvious/common sense/trivial. So are most of the problems in IT. Fred Brooks and Jerry Weinberg pretty much nailed down all the essential issues in IT project and personnel management more than 30 years ago; yet, amazingly, the problems haven’t all gone away! There is a profound lack of professional and institutional memory in IT; almost everyone who writes about IT project/personnel management (myself included) is looking for new ways to cast or explain the core issues in a touching hope that maybe this time someone will actually listen and fix them.
The Dead Sea effect is just the Peter Principle (or a corollary thereof). No, it isn’t. The Peter Principle is that a given person rises (is promoted) to her/his level of incompetence (I’m actually old enough to remember when ‘the Peter Principle’ first came out). This has nothing to do with promotion within the IT organization; it has to do with self-selected removal from that IT organization, not due to a lack of promotion or opportunity, but just because there are greener pastures elsewhere.
Not all IT shops are like this. I would certainly hope so. In fact, there are IT organizations where just the opposite occurs; the quality of the IT engineers is quite high, and engineers who are mediocre or disruptive either don’t get hired or don’t last long if they are. I worked in one such IT group (Pages Software) for five years. During that time, we had only one voluntary departure (the network admin); we had two others who were dismissed due to problems, and a few others who were (painfully) cut in downsizing. (On the other hand, here are some thoughts on why people would leave an outstanding (and lucrative) IT organization like Google.)
Not everyone ‘left behind’ is incompetent. Again, this syndrome doesn’t apply to all IT groups, and it doesn’t apply to the same extent to all IT groups. Turnover in IT personnel is common (though it can be reduced by intelligent management), and just because good engineers have left a given IT group doesn’t mean that the rest are, in fact, residue. What I’m talking about here is a very real syndrome, typically found in large corporations and government organizations, but it’s certainly not universal.
The IT hiring process is broken. Amen. Not only is the IT hiring process broken in many organizations, the entire approach to IT is often broken. It is rife with empire-building, ‘heroic’ project management, and an ‘interchangeable code monkeys’ mindset. As mentioned in the comments section below, Ruby Raley and I wrote an article for Cutter IT Journal that stated that an approach modeled after professional sport teams could well be far more effective, but no one has yet hired me to try it out. On the other hand, I would argue that this is to a large extent the approach we took as Pages, which is why we had such a great and effective IT group with so little turnover.
The problem is a failure of leadership. Again, amen. I wrote an entire book about that over a decade ago (The Art of ‘Ware, M&T Books, 1995), which I’m currently revising.
This doesn’t describe/encompass all the problems in professional IT shops. If it did, life would be much easier. Again, note that I’ve written a bit on the subject. I’m currently working on revised versions of two of my books (The Art of ‘Ware [Version 2.0] and Pitfalls of Modern Software Engineering) while writing a new one (Surviving Complexity), from which this posting on the “Dead Sea effect” is a very, very tiny extract.
Thanks for the great comments and the feedback; I wish I could get you folks to do this for all sections of all of my books.
For real-world examples of this phenomenon, just spend some time reading The Daily WTF (a site I highly recommend for all IT managers). In particular, their “Tales from the Interview” category gives some interesting insights from both sides of the hiring process.
Also, Alex Papadimoulis, who runs The Daily WTF, has put forth his own proposal for dealing with IT turnover. I especially like his concept of the Value Apex. Be sure to read his article; here’s my response to it ..bruce..
About the Author: bfwebster
Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com. Comments (102)
Trackback URL | Comments RSS Feed
Sites That Link to this Post
- Respect Capitalism » Blog Archive » The “Dead Sea” Effect in IT | April 12, 2008
- IT Hiring | April 13, 2008
- scalability.org » Blog Archive » An interesting bit on IT shops … | April 13, 2008
- Jonny’s Play Lunch | April 13, 2008
- links for 2008-04-14 « Simply… A User | April 13, 2008
- Ayoi’s » Blog Archive » Dead-sea-effect | April 14, 2008
- links for 2008-04-14 | April 14, 2008
- Dead Sea Effect at {inercia} | April 14, 2008
- Oso Rojo » Desalination | April 14, 2008
- links for 2008-04-15 - fractalnavel | April 14, 2008
- El efecto Mar Muerto y los departamentos de tecnología » El Blog de Enrique Dans | April 15, 2008
- El efecto Mar Muerto y los departamentos de tecnología [ENG] | April 15, 2008
- Kaizen Bits :: Dead Sea Effect | April 15, 2008
- El efecto Mar Muerto y los departamentos de tecnología « Foro de Tecnología y Negocio | April 15, 2008
- El efecto Mar Muerto y los departamentos de tecnología « nibarcomblog | April 15, 2008
- uglychart.com: a blog about stocks » Blog Archive » links for 2008-04-16 | April 15, 2008
- El efecto mar muerto « Diario de un Observador | April 17, 2008
- Is wetware such a misunderstood asset? « I see 0xDEAD disks | April 18, 2008
- los sueños de la razón / El semanal de anotaciones (primavera 2008, 5º domingo) | April 20, 2008
- el blog de jant » El efecto Mar Muerto y los departamentos de tecnología | April 21, 2008
- Top most Firing/ Secure IT companies in India - Team-BHP | April 21, 2008
- links for 2008-04-22 « D e j a m e S e r | April 22, 2008
- SuperUSER BLOG » Blog Archive » El efecto Mar Muerto y los departamentos de tecnología | April 23, 2008
- Blog de musica y otros cuentos » Blog Archive » El efecto Mar Muerto y los departamentos de tecnología | April 24, 2008
- The Wetware Crisis: the Expert Pool : Bruce F. Webster | April 26, 2008
- It makes business sense to… hire mediocre employees…? | Yousef Ourabi | April 26, 2008
- Some thoughts on “Up or Out” : Bruce F. Webster | April 29, 2008
- links for 2008-04-30 | Lazycoder | April 30, 2008
- Bolg - The Chris Blanc Weblog :: The Incidental Workforce | April 30, 2008
- The Dead Sea Effect: why would IT engineers leave Google? : Bruce F. Webster | May 5, 2008
- Credity » Blog Archive » The Wetware Crisis: the Dead Sea effect | May 6, 2008
- El efecto Mar Muerto y los departamentos de tecnología | BLOG USUARIO 5 | May 30, 2008
- Shapes & Lines » Blog Archive » …”The Dead Sea Effect” | June 12, 2008
- Anatomy of a runaway IT project : Bruce F. Webster | June 16, 2008
Before leaving IBM I tried to explain this process to my managers there over and over again. Here’s the funny thing: layoffs tended to make things worse. IBM had never done layoffs until the crisis of the mid-1990s. When they finally did it they offered amazingly lucrative packages to leave. Anyone with any ability to find a roughly comparable job leapt at the chance to take the package. The remaining employees where those that had either decided they were going to attempt to stay with the company for life or those that knew in their hearts that they weren’t worth what IBM was paying them and they’d never find a better deal elsewhere.
This of course reduced moral and the good programmers who were left could see what was going on, which only made the effect more obvious when the next round of voluntary layoffs came. Rather than make hard decisions and get rid of dead weight, the company abdicated its decision making power to the employees themselves and got the worst possible outcome. Instead of trimming the fat they allowed the meat to trim itself.
When I left software group for services I saw a different pattern but the same results. In services people were measured entirely by billable hours. The quality of your work only mattered if the customer was irate because you couldn’t get anything done. Stellar work was not valued any more than mediocre work. Therefore bonuses, raises, and promotions (when there were any) went to those that billed the most hours. Employees that were more effective and got more done in less time were undervalued. These people were the most likely to leave a system that didn’t properly value them and join a smaller company where there was room for there talent to be recognized and for them to be rewarded for their actual contribution.
In both cases the problem had the same root: an inability to properly recognize and value talent. Large organizations have to be especially disciplined when it comes to this or they’ll see their talent evaporate.
John:
Excellent observation; I hadn’t thought of the layoff issue, particularly when a buyout/severance package is involved.
Ruby Raley and I co-authored an article for the Cutter IT Journal, “The Longest Yard: Reorganizing IT for Success”. Our fundamental argument — which I think is supported by your comments — is that corporations need to treat IT more like a professional sports team and less like a manufacturing line. I may post it here. ..bruce..
The illogical conclusion is that one day fifth graders will have programmer trading cards and grown men will pour over pages of stats in order to build up their fantasy coding teams.
“I’ll trade you my Linus Torvalds for a your Guido van Rossum…”
Not only is the inability to recognize and reward talent severely lacking, but the perception that IT staff is merely a consequence and a drain on the budget is also rife throughout the industry. Managers just see that the IT engineers are always requesting new computers, servers, hardware, software licenses, training, and the budget has no place for a department that doesn’t contribute to the bottom line, but is a drain on it. The value of IT personnel is not there because those in decision-making positions are not aware of the long term consequences for untrained staffs and outdated equipment.
My fear is that this ignorance will have severe long term consequences on the ability of US corporations (and its government) to compete in a global economy. (We’re already falling behind here…)
Thanks for the article though – found it today through /.
Jason –
Excellent point, which gets to the heart of a fundamental challenge with IT that has been kicked around for years: how do you measure the value that IT brings to your organization? Tightly connected to that is whether IT is simply a “must have” like facilities and the HR department, or if IT provides a competitive edge.
Another problem that your comment touches on — and which is worthy of an entire post by itself — is the “penny wise, pound foolish” attitude regarding hardware and software. It makes no sense to spend a lot of money hiring and keeping an IT engineer and then penny-pinching on her/his development tools, yet I’ve seen that time and again. And that also tends to drive the best IT engineers away. ..bruce..
I agree and disagree with this post on a few points. I have worked in a position where the company was and to an extent is still like this. Talented people go…. but some leave who we are very happy to see the back of.
I’ve railed at management before about pulling out a crowbar and putting it into their wallet to start paying us market rates. Myself, I am an expert in what I do. I have designed and implemented multiple complete server farms, web and citrix. Done AD implementations as well as have designed complete antispam solutions for several large organisations that are clients of ours. I am proficient enough in spam management that the company that makes the package we use has had me as a keynote speaker at 2 forums and I have been invited to do it 4 times, including the offer to fly to another city and do the one there.
The assertion that those of us left behind are boobs with no brains is a bit incorrect. There are people with no brains left behind. And those of us gifted with good ones, we work our backsides off keeping the company afloat. Don’t forget that some people will keep a job out of loyalty, or because they have situations in their life that a move in jobs can destabilise. Having 2 small kids and my spouse not working full time means I can’t just up and move when I have some security where I am. And having immigrated to where I am now, until I was a citizen, moving jobs was even harder. Now that I am a citizen and it’s easier to move around…. management is finally coming to the table. But don’t tar us all with the same brush. There are incredibly talented unappreciated people out there who have a bit more loyalty than to jump ship because some management are clueless. It takes a long time, but that type of patience is rewarded a lot of times in the end.
I think to some extent many of the issues in todays IT workplace are caused by a lack of trust and vision in the right places; or to put it more simply; the guys making the important decisions often lack a certain level of vision and or insight, and don’t trust others who do have the vision. To some extend I suspect this is why Google is considered a great place to work by geeks, forget all the perks. If someone has a great idea, they’re given the ability to chase the idea down until it becomes more solid and easier to digest by those that perhaps don’t have the same vision within the organisation. The Google workers are trusted, and are essentially assumed to be smart and visionary and allowed the opportunity to shine.
In any other organisations, the decision makers (and pointy haired bosses) tend to value other other traits as being the most important, usually those who play to the ego’s or reinforce the leadership decisions that have been made in the past, or even perhaps some other totally random non-relevant criteria. Which means that those most highly valued by the organisation are often not those who bring the most (skill, vision, creative thinking, etc.) to it.
Just my 2 cents.
I work in Cambridge, UK, and occasionally high technology companies fold and people are made redundant, big names which have closed the facilities include Philips, ARM, Motorola, Nokia; sometimes a few years later when the technology recessions are over they start a new facility, or buy a company which has one here.
In the UK a company is required to offer voluntary redundancy to their staff if they are reducing headcount, and the offer is usually quite good when trouble first looms. I always advise my friends to be the first to sign up, as the initial offers to reduce headcount are the most generous. If the company still struggles and needs to shed more staff, the 2nd wave get a noticeably worse settlement. If the company folds the people who are left only get the statutory settlement which is typically what they’re owed plus maybe one months salary if that; sometimes they receive nothing!
Being the first out of the door also means being the first to hit the job market and find a new job; being the last means you’re just one of many, and the market is then swamped with people with similar skills.
I have to disagree with the basic premise underlying your post: that the majority of jobs call for top-talent developers. My time in the industry (since the 80s) shows that the majority of development positions are essentially intermediate level coding positions maintaing existing systems. For a new team member, these jobs have a a significant ramp up time learning the ins and outs of an fully developed and in production system, but then fall into a pattern of maintenance and incremental changes.
Try to find people willing to work existing systems — data cleanup, data transfers, system integration, reports, process changes and screen tweaks — day in and day out. Everyone is either too junior and would end up destabilizing an existing system or too “talented” and would rather design the next great thing. It’s these last group of people, who come in as top talent, realize they don’t get to spend all their time playing designer with the latest technology, and then leave when they get bored, who are truly the problem.
And these are are YOUR people? Top coders who jump ship once they realize that their job is going to be work rather than play. But only after they’ve taken six months getting up to speed on an existing application and after taking the time and effort of the rest of the team to get to there. And all the while insisting how things would be so much easier if they could be allowed to just recode the entire system using an aspect oriented, dependency injected, MVC design along with a meta-hibernated data layer. These people probably are truly talented, it’s just that they won’t accept as their long term job the real work that needs to be done over the work they want to do.
Give me an ocean of “salt” over your top-talent. I don’t get to create new systems every day, but each and every day I DO have to keep existing systems in real work places running smoothly as the work place changes around them.
Will:
Go look at my post on TEPES. I fully agree that professionalism trumps talent. In fact, I once got turned down for an IT management job — where there were a couple of well-established prima donna programmers — for making that clear. But you’re mischaracterizing the situation I’m describing (and you’re describing a largely distinct problem).
I don’t say and I’m not saying that everyone has to be brilliant. What I am saying is that everyone should be competent and professional. And the problem with some organizations is that the residue tends to be marginal in both aspects. The organization keeps them not because they’re quietly doing a good job but because (a) the organization isn’t really able to judge how good a job they’re doing and/or (b) the organization has a hard time finding replacements.
I didn’t start out as a chief architect right out of college. I did a lot of associate software engineer jobs at an associate software engineer level. I worked my way up through my professionalism and my competency. Talented entry-level personnel — under the direction of someone competent — can do a great job on the maintenance tasks that you describe. They’ll eventually want to move on, but you can use them to supervise their replacements. Think of it a farm system. ..bruce..
The only problem with the analogy of the “farm system” though is that individuals (like myself) who have advanced well past the basic levels of competency, and who would be interested in moving into a supervisory position to teach and train others are not really seen as a valuable resource in their current positions (because they will be leaving) or in other organizations (because they don’t know [b]that[/b] system.
A very interesting way of describing operational and subsequently mental slowdown and breakdown.
I have been descibing this very phenomenon as the “PublicSectorization” disease which is: Any organization be it an IT dept. a marketing dept., a group of employees what have you, has a very private critical mass. Each group has a very different critical mass dependent on many parameters such as structure, ease of communication , innovation , etc. but very specific to the group itself. Once that particular organization reaches its specific critical mass suddenly all work starts churning out at Public Sector Employee speeds, which is extremely slow, mostly unresponsive and not always of expected quality.
Will:
Why is such a high percentage of development activity in your company purely maintenance?
Have you hired an insufficient number of “the talent” in the past?
Easily maintained, truly extensible, agile codebase’s are possible. Note, this is orthogonal to over-engineered bells-and-whistles “play” coding, to which you alure.
You need to know what such a codebase looks like before you can even begin to successfully hire anyone to produce code of that form.
Positive reinforcing feedback?
Part of the problem is the inherent tendency (rising with talent, I believe) for individualism and libertarianism (small-l) on the part of IT people.
The more competent ones tend to be so for the basic reason that they are more capable of thinking for themselves, so don’t have their identities tied up with any specific organization, and they also tend to be far less willing to put up with the amount of crap that larger organizations generate as a matter of course:
I was at an organization where the mandate came from on high — no more polos, only button down shirts were allowed.
As though this is going to help the organization HOW?
Said same organization was also in the process of creating a defacto hostile working environment for males by implementing ludicrous sexual harrassment rules — basically, you were screwed if anyone made a complaint — no proof required, no questions asked.
We’re not talking about real harassment where someone actually did anything — we’re talking about ANYONE who took offense at anything, no matter how trivial or stupid.
Needless to say, the smarter people were swiftly to be found elsewhere.
O Bloody Hell:
Great comment and I believe spot on. Take a look at my first (and so far only) “Negotiations and Lovesongs” posting for my abstract pattern of Suits v. Geeks v. Users. It’s deliberately overdrawn, but with a foundation of truth. ..bruce..
There’s one other thing at work here, when talented and motivated engineers get married and/or have kids, they turn into salt crystals. They feel they can no longer take risks and settle for not making waves and working for a steady paycheck. I’d go so far as to say that if I were running a small coding shop, and had the choice between a crew of 30-year-old married engineers, or a crew of 50-year-old single or divorced engineers, I’d definitely choose the latter. They’ll have fewer outside restrictions and inhibitions imposed, and will be able to do their best work , or leave if they don’t like it (which would be a catalyst for introspection and change at the company). I suspect the reason software companies seem to prefer younger employees has everything to do with marital status and focus at work and relatively little to do with gray hair and any difference in energy or cognitive abilities. The best coders I’ve ever met were older and happily unmarried.