Subscribe via RSS Feed

Anatomy of a runaway IT project

June 16, 2008 28 Comments

The following document is the actual text — carefully redacted — of a memo I wrote some time back [i.e., several years ago] after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered.

Note that the “ABC” consultants were a small part of the overall project team and had been brought in relatively late by “BigFirm” in an attempt to get the “FUBAR” project into production; they neither initiated nor managed the project. [NOTE for those of you who have written or done Google searches: “Bob Winsom”, like all the other names in the memo as transcribed below, is a pseudonym.]

========================================

CONFIDENTIAL MEMORANDUM — EYES ONLY

Over the past two weeks, I’ve conducted confidential off-site group interviews with all of the ABC consultants working on the FUBAR project. I did this at [ABC manager’s] request, after a few of these consultants spoke privately about FUBAR with him. The feedback was consistent and raises serious doubts about whether the FUBAR project, as currently pursued, can ever yield a successful production deployment.

This report groups those comments into several broad areas. It is relatively unfiltered and extremely direct (no withholding). It represents the private comments of ABC consultants who have little to gain and possibly much to lose by being so blunt. These are not the whinings of purists picking nits. These are the grounded assessments of top-notch IT professionals who have among them a century or two of experience bringing projects to completion — particularly those involving [specific IT] technology — and who are down in the FUBAR trenches every day.

QUALITY OF WORK AND EFFORT

ISSUE: Several consultants said — and the rest pretty much agreed — that far too many of the deliverables, artifacts, and activities (e.g., algorithms, source code, system configuration, design/architecture documents, testing, defect tracking, scheduling etc.) are substantially below any acceptable professional standards and represent a profound threat to FUBAR ever going into production.

EXAMPLES: The code base is very fragile. A lot of it is bad old code that BigFirm didn’t have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value ‘3’, a constant named ‘ninety_eight’). Builds take all night. App releases don’t run acceptably, if at all, in a production environment. Developers check in files that won’t even compile.

RISKS: The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project.

PROJECT PLANNING AND EXECUTION

ISSUE: Project planning and execution is all to often poor or missing completely. Milestone dates, usually unrealistic if not impossible, are based on political considerations or wishful thinking, not bottom-up grounding. Necessary and useful activities are delayed or canceled with the justification “We have no time for that”, but the project phase ends up taking as long or longer than if the activities had been carried out. Dates are set, but nobody scrambles until the last minute. Risks are not actively tracked or managed.

EXAMPLES: Count how many times FUBAR ever produced a production-quality deliverable on anything close to a scheduled date. Even the current effort probably requires a year to get something into production, but the schedule says four months. Managers create work tasks, but then never track progress or completion. One ABC consultant created a risks document; Bob Winsom [BigFirm’s FUBAR project manager] took it over, and no one has seen it since.

RISKS: FUBAR is massively late. You lose credibility and influence.

QUALITY ASSURANCE AND PROCESS

ISSUE: Quality assurance appears to be low-priority concept within the FUBAR project. In the opinion of several consultants, the current person in charge of it is not particularly strong or competent. There appears to be a systemic inability to establish good testing criteria and methods to gauge FUBAR’s progress toward production. What software lifecycle process is in place is often not followed. No independent group or person acts as the ‘gatekeeper’ to judge acceptability and control release into production.

EXAMPLES: [Key process] calculation — the core of BigFirm’s business and profits — was being (and may still be) done incorrectly in FUBAR; it had never been previously checked for correctness through all these years. Likewise, performance expectations have been based on the presumption of FUBAR distributed over multiple systems, processors, and threads, yet no one ever tested to see if those implementations would work until recently — and they didn’t. The build environment needs to be overhauled. The defect tracking process is poor, particularly the practice of writing up defects not against the current release but the release in which the defect is scheduled to be fixed — so as to keep the number of defects down for the current release.

RISKS: BigFirm leaves itself open to potential liabilities, not to mention crippling its own core business. In the meantime, the effort to transition directly into the Rational Unified Process (RUP) is not being given sufficient time and will likely grind development to a halt.

ARCHITECTURE

ISSUE: FUBAR doesn’t have a viable, consistent architecture. The irony is that FUBAR itself is not a big, complex problem; it is a relatively straightforward problem that has been made big, complex, and possibly unsolvable in the current implementation. Initiatives to rearchitect are started, abandoned due to “schedule pressure”, then restarted months later.

EXAMPLES: The project has gone through a series of architects, who have either left or been asked to leave. While they are here, they usually are neither listened to nor given the authority to be an architect. Technical decisions are made by people lacking the background, such as Bob Winsom, who may fancy himself an architect and was quoted as saying, “I haven’t found an architect I like yet.”

RISKS: FUBAR never stabilizes enough to go into production for any length of time, or if it does, proves to be extremely difficult to support or enhance.

APPLICATION PERFORMANCE

ISSUE: FUBAR was never properly architected and designed for the performance required. There is a current effort to increase performance after the fact, but the implementation makes that impossible. To make things worse, developers are having to scale the performance of and debug a seriously flawed application at the same time, making it very hard to stabilize the application.

EXAMPLES: Two consultants rewrote the 140,000 lines of [original obscure language] into 4200 lines of Java. The Java version runs as fast on a laptop PC as the original version runs on a high-powered UNIX server.

RISKS: Despite heroic efforts (that will probably make the application even more difficult to modify and support) and lots of hardware, FUBAR will probably reach some fraction of the currently-desired performance — possible as little as 15% to 20% of that required, possibly as much as 80% — and then go no further.

STAFFING

ISSUE: Many of the people involved in FUBAR — developers, testers, team leads, managers — lack the qualifications, insight, or experience to make FUBAR a success. The project is overstaffed for the actual complexity of FUBAR, possibly for political reasons (i.e., promotion/stature based on the number of people supervised).

EXAMPLES: Many of the examples listed above reflect this problem.

RISKS: This problem leads in part to all the issues previously listed: poor quality of work, poor quality assurance, poor scheduling and delivery, poor architecture, poor application performance. Besides the potential failure of FUBAR itself, this issue tends to be self-intensifying — that is, the qualified people become frustrated and leave (or are hard to recruit in as replacements), while those less qualified or capable stay around. [A reference to the “Dead Sea effect” written many years ago.]

[BrandName team management approach] PRINCIPLES

ISSUE: Mid-level management tells the developers that mood, sincerity, and commitment are everything, and that with them “we can accomplish anything.” At the same time, the principle of granting sincerity appears to be used to justify a lack of accountability and consequences.

EXAMPLES: Repeated statements in team meetings, one-on-ones, and so on.

RISKS: Loss of credibility. Such assertions don’t hold water. I can be in a great mood and have a team of very sincere and committed people, but if we try to build a commercial airliner without the proper expertise, requirements, engineering, materials, and testing, the plane will crash and people will die, assuming it ever gets built and off the ground (which is extremely unlikely). The fallacy that software is somehow different is just that — a fallacy, and one that costs corporations millions (if not billions) of dollars a year in missed schedules and failed projects. When it comes to engineering, sincerity and commitment, while important, can never substitute for expertise and quality of work.

INTELLECTUAL HONESTY

ISSUE: There isn’t enough intellectual honesty within the FUBAR project. Managers reject or explain away bad news and real problems, looking instead for people who will tell them what they want to hear.

EXAMPLES: Several developers and team leads have sought to escalate these issue and concerns up the management chain, but the issues appear to always get stopped, usually at Bob Winsom. [The “thermocline of truth“, with a very discrete boundary.] The FUBAR project is represented as something that has never been done before, and the staff as a world-class development team.

RISKS: The lack of intellectual honesty in project management is a form of codependency and enabling that is all too easy to fall into. Unfortunately, reality eventually intrudes, and when it does, you run the very real risk of looking dishonest or incompetent.

CLOSING REMARKS

As I said, this is a very blunt (and very confidential) memo. It represents the opinions, experiences, and observations of these ABC consultants, and there are undoubtedly points with which you take issue or disagree. Do not let that blind you to the fundamental reality that there are some profound problems and flaws with the FUBAR project that will not go away until the project team acknowledges and addresses them. Indeed, it will be hard enough to make them go away even then.
========================================

Kind of scary, isn’t it? The interesting part was that BigFirm had implemented, corporate-wide, a “team management” methodology (from an outside firm) that was based on “mood, sincerity, and commitment”. As an overall corporate management approach, it might well have been effective; I just don’t know. But BigFirm thought that it would also solve their IT problems.

Nope, it didn’t. ..bruce..

[Speaking of project failure — I have my first three “Surviving Complexity” columns up at Baseline, talking about IT project metrics, why they’re so tough to define, and one possible approach.]

[UPDATED 06/25/08: If you think the project above is bad, take a look at this one.]

About the Author:

Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com.

Comments (28)

Trackback URL | Comments RSS Feed

Sites That Link to this Post

  1. The Wetware Crisis: the Dead Sea effect : Bruce F. Webster | June 16, 2008
  2. The Wetware Crisis: the Thermocline of Truth : Bruce F. Webster | June 16, 2008
  3. nerdd.net | news and opinion | June 17, 2008
  4. The Dead Sea Effect | FlishFun.com | June 17, 2008
  5. Anatomy of a runaway IT project | Project Server Blogs | June 17, 2008
  6. Das Ganze » Project Managing | June 17, 2008
  7. Vast and Infinite » Blog Archive » Software: The Agony and the Ecstasy | June 17, 2008
  8. Anatomy of a Runaway IT Project | Bruce F. Webster | Voices | AllThingsD | June 18, 2008
  9. onda kada mešate babe i žabe… | Dan u životu (srpske) kompanije | June 18, 2008
  10. Something about Gene » Blog Archive » New Firefox, runaway projects | June 18, 2008
  11. Arjan`s World » LINKBLOG for June 18, 2008 | June 18, 2008
  12. links for 2008-06-18 « ?????? | June 18, 2008
  13. FUBAR | ???? | June 19, 2008
  14. links for 2008-06-20 « Dan Creswell’s Linkblog | June 19, 2008
  15. Looking back at a major/failed IT project | Karim Vaes | June 19, 2008
  16. Project from Hell « Project Failures | June 25, 2008
  17. Bruce’s Blog / Anatomy of a runaway IT project : Bruce F. Webster | June 28, 2008
  18. IT Project Failures mobile edition | July 23, 2008
  19. Bookmarks about Projectmanagement | December 27, 2008
  20. links for 2008-06-19 : Elemental Links | July 25, 2010
  21. Project Managing | HeyLookAtAllThisGreatStuff | August 26, 2012
  22. Septic code: why some large IT projects never go into production : Bruce F. Webster | September 12, 2013
  23. £10 billion ($16 billion) IT failure — yep, it’s the UK again : Webster & Associates LLC | September 18, 2013
  24. Obamacare: descent into the maelstrom [UPDATED] : And Still I Persist… | October 9, 2013
  25. Obamacare, IT, and magic thinkiing : And Still I Persist… | October 21, 2013
  26. Obamacare and the Unstopped Project : And Still I Persist… | November 13, 2013
  1. fred says:

    Very insightful, with some ideas that I could relate to.

  2. I think the admin of this site is in fact working hard for his website,
    as here every information is quality based information.

Leave a Reply

You must be logged in to post a comment.