Bruce F. Webster

My newest Baseline column is up: “Lies, Damned Lies, and Project Metrics (part 3)“. In it, I wrap up my discussion on IT project metrics, outlining a possible approach using instrumentation and heuristics.  Go check it out.  ..bruce..

The following document is the actual text — carefully redacted — of a memo I wrote some time back [i.e., several years ago] after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered.

Note that the “ABC” consultants were a small part of the overall project team and had been brought in relatively late by “BigFirm” in an attempt to get the “FUBAR” project into production; they neither initiated nor managed the project. [NOTE for those of you who have written or done Google searches: "Bob Winsom", like all the other names in the memo as transcribed below, is a pseudonym.]

========================================

CONFIDENTIAL MEMORANDUM — EYES ONLY

Over the past two weeks, I’ve conducted confidential off-site group interviews with all of the ABC consultants working on the FUBAR project. I did this at [ABC manager's] request, after a few of these consultants spoke privately about FUBAR with him. The feedback was consistent and raises serious doubts about whether the FUBAR project, as currently pursued, can ever yield a successful production deployment.

This report groups those comments into several broad areas. It is relatively unfiltered and extremely direct (no withholding). It represents the private comments of ABC consultants who have little to gain and possibly much to lose by being so blunt. These are not the whinings of purists picking nits. These are the grounded assessments of top-notch IT professionals who have among them a century or two of experience bringing projects to completion — particularly those involving [specific IT] technology — and who are down in the FUBAR trenches every day.

QUALITY OF WORK AND EFFORT

ISSUE: Several consultants said — and the rest pretty much agreed — that far too many of the deliverables, artifacts, and activities (e.g., algorithms, source code, system configuration, design/architecture documents, testing, defect tracking, scheduling etc.) are substantially below any acceptable professional standards and represent a profound threat to FUBAR ever going into production.

EXAMPLES: The code base is very fragile. A lot of it is bad old code that BigFirm didn’t have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value ‘3′, a constant named ‘ninety_eight’). Builds take all night. App releases don’t run acceptably, if at all, in a production environment. Developers check in files that won’t even compile.

RISKS: The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project.

PROJECT PLANNING AND EXECUTION

ISSUE: Project planning and execution is all to often poor or missing completely. Milestone dates, usually unrealistic if not impossible, are based on political considerations or wishful thinking, not bottom-up grounding. Necessary and useful activities are delayed or canceled with the justification “We have no time for that”, but the project phase ends up taking as long or longer than if the activities had been carried out. Dates are set, but nobody scrambles until the last minute. Risks are not actively tracked or managed.

EXAMPLES: Count how many times FUBAR ever produced a production-quality deliverable on anything close to a scheduled date. Even the current effort probably requires a year to get something into production, but the schedule says four months. Managers create work tasks, but then never track progress or completion. One ABC consultant created a risks document; Bob Winsom [BigFirm's FUBAR project manager] took it over, and no one has seen it since.

RISKS: FUBAR is massively late. You lose credibility and influence.

QUALITY ASSURANCE AND PROCESS

ISSUE: Quality assurance appears to be low-priority concept within the FUBAR project. In the opinion of several consultants, the current person in charge of it is not particularly strong or competent. There appears to be a systemic inability to establish good testing criteria and methods to gauge FUBAR’s progress toward production. What software lifecycle process is in place is often not followed. No independent group or person acts as the ‘gatekeeper’ to judge acceptability and control release into production.

EXAMPLES: [Key process] calculation — the core of BigFirm’s business and profits — was being (and may still be) done incorrectly in FUBAR; it had never been previously checked for correctness through all these years. Likewise, performance expectations have been based on the presumption of FUBAR distributed over multiple systems, processors, and threads, yet no one ever tested to see if those implementations would work until recently — and they didn’t. The build environment needs to be overhauled. The defect tracking process is poor, particularly the practice of writing up defects not against the current release but the release in which the defect is scheduled to be fixed — so as to keep the number of defects down for the current release.

RISKS: BigFirm leaves itself open to potential liabilities, not to mention crippling its own core business. In the meantime, the effort to transition directly into the Rational Unified Process (RUP) is not being given sufficient time and will likely grind development to a halt.

ARCHITECTURE

ISSUE: FUBAR doesn’t have a viable, consistent architecture. The irony is that FUBAR itself is not a big, complex problem; it is a relatively straightforward problem that has been made big, complex, and possibly unsolvable in the current implementation. Initiatives to rearchitect are started, abandoned due to “schedule pressure”, then restarted months later.

EXAMPLES: The project has gone through a series of architects, who have either left or been asked to leave. While they are here, they usually are neither listened to nor given the authority to be an architect. Technical decisions are made by people lacking the background, such as Bob Winsom, who may fancy himself an architect and was quoted as saying, “I haven’t found an architect I like yet.”

RISKS: FUBAR never stabilizes enough to go into production for any length of time, or if it does, proves to be extremely difficult to support or enhance.

APPLICATION PERFORMANCE

ISSUE: FUBAR was never properly architected and designed for the performance required. There is a current effort to increase performance after the fact, but the implementation makes that impossible. To make things worse, developers are having to scale the performance of and debug a seriously flawed application at the same time, making it very hard to stabilize the application.

EXAMPLES: Two consultants rewrote the 140,000 lines of [original obscure language] into 4200 lines of Java. The Java version runs as fast on a laptop PC as the original version runs on a high-powered UNIX server.

RISKS: Despite heroic efforts (that will probably make the application even more difficult to modify and support) and lots of hardware, FUBAR will probably reach some fraction of the currently-desired performance — possible as little as 15% to 20% of that required, possibly as much as 80% — and then go no further.

STAFFING

ISSUE: Many of the people involved in FUBAR — developers, testers, team leads, managers — lack the qualifications, insight, or experience to make FUBAR a success. The project is overstaffed for the actual complexity of FUBAR, possibly for political reasons (i.e., promotion/stature based on the number of people supervised).

EXAMPLES: Many of the examples listed above reflect this problem.

RISKS: This problem leads in part to all the issues previously listed: poor quality of work, poor quality assurance, poor scheduling and delivery, poor architecture, poor application performance. Besides the potential failure of FUBAR itself, this issue tends to be self-intensifying — that is, the qualified people become frustrated and leave (or are hard to recruit in as replacements), while those less qualified or capable stay around. [A reference to the "Dead Sea effect" written many years ago.]

[BrandName team management approach] PRINCIPLES

ISSUE: Mid-level management tells the developers that mood, sincerity, and commitment are everything, and that with them “we can accomplish anything.” At the same time, the principle of granting sincerity appears to be used to justify a lack of accountability and consequences.

EXAMPLES: Repeated statements in team meetings, one-on-ones, and so on.

RISKS: Loss of credibility. Such assertions don’t hold water. I can be in a great mood and have a team of very sincere and committed people, but if we try to build a commercial airliner without the proper expertise, requirements, engineering, materials, and testing, the plane will crash and people will die, assuming it ever gets built and off the ground (which is extremely unlikely). The fallacy that software is somehow different is just that — a fallacy, and one that costs corporations millions (if not billions) of dollars a year in missed schedules and failed projects. When it comes to engineering, sincerity and commitment, while important, can never substitute for expertise and quality of work.

INTELLECTUAL HONESTY

ISSUE: There isn’t enough intellectual honesty within the FUBAR project. Managers reject or explain away bad news and real problems, looking instead for people who will tell them what they want to hear.

EXAMPLES: Several developers and team leads have sought to escalate these issue and concerns up the management chain, but the issues appear to always get stopped, usually at Bob Winsom. [The "thermocline of truth", with a very discrete boundary.] The FUBAR project is represented as something that has never been done before, and the staff as a world-class development team.

RISKS: The lack of intellectual honesty in project management is a form of codependency and enabling that is all too easy to fall into. Unfortunately, reality eventually intrudes, and when it does, you run the very real risk of looking dishonest or incompetent.

CLOSING REMARKS

As I said, this is a very blunt (and very confidential) memo. It represents the opinions, experiences, and observations of these ABC consultants, and there are undoubtedly points with which you take issue or disagree. Do not let that blind you to the fundamental reality that there are some profound problems and flaws with the FUBAR project that will not go away until the project team acknowledges and addresses them. Indeed, it will be hard enough to make them go away even then.
========================================

Kind of scary, isn’t it? The interesting part was that BigFirm had implemented, corporate-wide, a “team management” methodology (from an outside firm) that was based on “mood, sincerity, and commitment”. As an overall corporate management approach, it might well have been effective; I just don’t know. But BigFirm thought that it would also solve their IT problems.

Nope, it didn’t. ..bruce..

[Speaking of project failure -- I have my first three "Surviving Complexity" columns up at Baseline, talking about IT project metrics, why they're so tough to define, and one possible approach.]

[UPDATED 06/25/08: If you think the project above is bad, take a look at this one.]

My newest Baseline column is up: “Lies, Damned Lies, and Project Metrics (part 2)“. In it, I talk about why it’s so hard to apply metrics to IT project management and begin to suggest an approach.  Go check it out.  ..bruce..

In my earlier post on the “thermocline of truth“, I wrote:

Second, IT engineers by nature tend to be optimists, as reflected in the common acronym SMOP: “simple matter of programming.” Even when an IT engineer doesn’t have a given subsystem completed, he tends to carry with him the notion that he whip everything into shape with a few extra late nights and weekends of effort, even though he may actually face weeks (or more) of work. (NOTE: my use of male pronouns is deliberate; it is almost always male IT engineers who have this unreasonable optimism. Female IT engineers in my experience are generally far more conservative and realistic, almost to a fault, which is why I prefer them. I just wish they weren’t so hard to find.)

Now, a post over at the Wall Street Journal cites what I think is a more controversial (and harder to support) claim — by a female VP of Engineering — that female programmers tend to write clearer and better-documented code than male programmers:

Emma McGrattan, the senior vice-president of engineering for computer-database company Ingres–and one of Silicon Valley’s highest-ranking female programmers–insists that men and women write code differently. Women are more touchy-feely and considerate of those who will use the code later, she says. They’ll intersperse their code–those strings of instructions that result in nifty applications and programs–with helpful comments and directions, explaining why they wrote the lines the way they did and exactly how they did it.

The code becomes a type of “roadmap” for others who might want to alter it or add to it later, says McGrattan, a native of Ireland who has been with Ingres since 1992.

Men, on the other hand, have no such pretenses. Often, “they try to show how clever they are by writing very cryptic code,” she tells the Business Technology Blog. “They try to obfuscate things in the code,” and don’t leave clear directions for people using it later. McGrattan boasts that 70% to 80% of the time, she can look at a chunk of computer code and tell if it was written by a man or a woman.

I’m not sure that I could make the same claim without some serious research into a broad range of coding samples. But the article is worth reading for the comments that follow it, which (as you can imagine) are quite intense.  ..bruce..

As I’ve mentioned here before, I’m writing a book called Surviving Complexity. Many of my posts here at this website have adapted from materials I’m writing for that book.

Well, now I’ve been hired by Ziff Davis Enterprises to write a weekly column on IT Management for the online version of Baseline. That column is titled “Surviving Complexity” and, again, draws upon work I’m doing for the book. My first column is up: “Lies, Damned Lies, and Metrics (Part I)”; here’s the opening paragraph:

When Capers Jones published Assessment and Control of Software Risks (Yourdon Press, 1994), he identified the most serious software risk in IT projects as “Inaccurate Metrics,” and the second most serious software risk as “Inadequate Measurement”. I remember being startled when I first read that back in 1995—they certainly weren’t what I would have chosen—and other authorities in the field criticized his choices. Yet, in the intervening years, I have moved closer and closer to Jones’ point of view.

I’ll still be writing here, both with materials relevant to Surviving Complexity and with my on-going revisions to The Art of ‘Ware. But please feel free to check out the new column. Thanks. ..bruce..