What is interesting about Holmgren’s hire is that it is modeled after Bill Parcells role at Miami – The Football Czar.  He’s not the head coach and he’s not the GM (who usually handles personnel).  Instead, he is something else.  It is a role that leverages his expertise as a position coach, a head coach and a GM.  One that sees the bigger picture and is able to evaluate players AND coaches from a fresh and more independent perspective.  It is a position created to drive the “rebuilding” of a program – something Miami and Cleveland badly need.  In Parcells’ case, he took an 1-15 team and got it into the playoffs the next year with an 11-5 record.  Part of the Parcells formula is to bring in a core of coaches and players that he trusts and who know his systems, both offensively and defensively.

Be sure to read Curran’s entire post, plus the robust debate in the comments that follow it.

Of course, Ruby and I also used the football analogy a few years ago in suggesting a radically different approach to IT organization and leadership. As we wrote:

The bad news is that, unlike in football, the business and IT sides of a firm don’t always agree on what constitutes a ‘victory’ (even though both sides can usually agree on what a ‘loss’ is, at least in cases of total or significant project failure). Indeed, sometimes they cannot fully agree on what the game is.

Curran’s IT Czar role could in theory solve that; I like his proposals and approach, and, of course, I like the sports team analogy.

The problem is, it’s hard to see how upper management would treat or perceive it as being any different from the current CIO role. Both as a consultant and as an expert witness, I’ve had lots of opportunity to see just how  constrained the CIO slot can be in large organizations. A lot of the blame for that rests squarely upon the CEO and other CxO leaders, both in terms of whom they select for the CIO job and how they define/constrain that job. CIOs tend to be selected from business-types with some technical background, as opposed to technical types with some business background. Curran’s analogy (e.g., Bill Parcells) suggests that the czar be someone who has been down in the trenches (e.g., position and head coach) and knows what day-to-day IT development requires, but that’s not who usually gets picked.

The closest model to an IT Czar that I’ve seen that has worked were the various corporate Y2K ‘czars’ who were appointed 10+ years ago to save the officers and directors from any Y2K liability, and thus were given lots of power and pretty free rein. (A joke from those times — Q: What’s the difference between a terrorist and a corporate Y2K director? A: You can negotiate with the terrorist.) An IT czar appointed to ‘turn around’ an organization’s IT efforts would have to be given that same power and freedom. That is not likely to happen unless, as with Y2K, the corporate officers and directors see themselves at risk — professionally, legally, financially and/or personally. Otherwise, as noted above, the “IT Czar” would be just another CIO.  ..bruce..

Here in Part II, I’ll talk about some of the well-established maxims and heuristics of complex systems development, and how they apply to legislation in general and to HR 3200 in particular. (More after the jump.)

Gall

As far as I can tell, John Gall — in his out-of-print book Systemantics (1976)– was the first to observe in print that

A complex system that works is found to have invariably evolved from a simple system that worked. (p. 80, 1978 paperback edition).

Immediately after, he observes that:

A complex system designed from scratch never works and cannot be made to work. You have to started over,  beginning with a working simple system.

My co-blogger (over at ASIP) Bruce Henderson puts this another way:

Start out stupid, and work up from there.

There is large room for differing arguments here as to just where HR 3200 fits in, for several reasons.

First, HR 3200 isn’t “designed from scratch.” As noted in Part I, many sections of HR 3200 are modifying various existing laws and regulations, such as the Internal Revenue Code, the Public Health Service Act, Employee Retirement Income Security Act, the Social Security Act, and the United States Code.

However, leveraging upon and modifying several existing systems is not the same as building a “simple system that works” and evolving it into a complex system that works. I can create a large, complex piece of software that calls upon and even modifies existing systems and libraries — but that doesn’t necessarily mean I’m evolving something from a “small, simple system that works”. This is especially true when I’m pulling together from several disjoint or unrelated systems (such as those listed above).

Second, legislation is more robust than software, for exactly the differences outlined in part I, namely that legislation is executed by people rather than machines and operating systems. If I create an ill-formed piece of software, there’s a good chance it won’t even compile (or interpret); if it does, then it may run into linking or integration errors; and if it gets past those, it may crash, lock up, or behave bizarrely upon execution.

If, however, I create an ill-formed piece of legislation, it can be (and often is!) be put into practice, with various human either officially or unofficially working around the defects to make it “work”. Of course, that ‘deployment’ of the legislation may end up drifting or even veering sharply from the stated or actual intent of the legislation. (In a way, this is reminiscent of the early PL/1 compilers that would, upon encountering a syntax error, make a best guess as to what you might have meant to write and compile that instead.)

Courts can shift this ‘deployment’ in both directions. They may “find” meaning or functionality in the law never contemplated or even explicitly disavowed by those who crafted and voted for the legislation, or they may prohibit some portion of explicit functionality due to conflicts with the Constitution, prior judicial rulings, or simply their own judgment.  As noted in Part I, judges don’t always agree with one another, either, so whether a given piece of legislation (or a subportion thereof) is upheld, modified, or rejected entirely depends upon which courts or individual judges end up reviewing it.

Third, there are serious and compelling arguments as to how well the current government health care programs (such as Medicare and the VA hospital system) work, not to mention the government systems modified and relied upon by HR 3200 (such as the IRS and Social Security). While you may argue with Gall’s maxims above, I know of no serious systems designer who will state that it is possible to build a large, complex system that works from complex systems that work poorly, if at all. The quality of your original and leveraged systems provides an upper bound on the quality of your final system. To believe otherwise is to succumb to wishful thinking.

Maier and Rechtin

In The Art of Systems Architecting by Mark W. Maier and Eberhardt Rechtin (2002), the authors take a cross-discipline approach to systems architecting, including talking specifically about social systems in Chapter 5. The following passage from that chapter is of particular relevance to the overall purpose of HR 3200 (all emphasis in the original):

The first insight, which might be called the four whos, asks four questions that need to be answered as a self-consistent set if the system is to succeed economically; namely, who benefits? who pays? who provides? and, as appropriate, who loses?

The political arguments raging over HR 3200 are exactly over those four questions. In fact, Maier and Rechtin themselves foresaw those arguments, since they go on to use health care as an example:

Example: serious debates over the nature of their public health services are underway in many countries, triggered in large part by the technological advances of the last few decades. These advances have made it possible for humanity to live longer and in better health, but the investments in those gains are considerable. The answer to the four whos are at the crux of the debate. Who benefits — everyone equally at all levels of health? Who pays — regardless of personal health or based on need and ability to pay? Who provides — and determines cost to the user? Who loses — anyone out of work or above some risk level, and who determines who loses?

The problem with HR 3200 and with the arguments put forth to date on its behalf is that they have not systematically and credibly addressed those four questions. In fact, those arguing in support of HR 3200 and health care reform in general have often given contradictory answers to those four questions, undermining their own credibility, given ammo to their opposition, and (justifiably) undermining public support for HR 3200.

Along those lines, the authors also note that in architecting social systems, you face not just the constraints of normal system design — risk, performance, schedule, and cost — but two more: perception vs. facts. They go on to say:

Social systems have generated a painful design heuristic: it’s not the facts, it’s the perception that counts. Some real-world examples: . . .

• One of the reasons that health insurance is so expensive is that health care is perceived by employees as nearly “free” because almost all its costs are paid for either by the the employee’s company or the government. The facts are that the costs are either passed on to the consumer, subtracted from wages and salary, taken as a business deduction against taxes, or all of the above. There is no free lunch.

Again, with great relevance to the current debate over HR 3200 and the whole approach of the House over health care reform, the authors state:

Like it or not, the architect must understand that perceptions can be just as real as facts, just as important in defining the system architecture, and just as critical in determining success. As one heuristic states: the phrase, ‘I hate it’, is direction. There have even been times when, in retrospect, perceptions were “truer” than facts which changed with observer, circumstance, technology, and better direction. . . . In the end, it is a matter of achieving a balance of perceived values. The architect’s task is to search out that area of common agreement that can result in a desirable, feasible system.

Maier and Rechtin end Chapter 5 with some heuristics they consider specific to social systems. Several are those already cited above, but here are a few additional ones (my comments are in brackets):

Success is the eye of the beholder [i.e., the US public] (not the architect [i.e., Congress]).

Don’t assume that the original statement of the problem [e.g., "45 million uninsured"] is necessarily the best, or even the right one. (Most customers would agree.)

In social systems, how you do something may be more important than what you do. (A sometimes bitter lesson for technologists [and Congress] to learn.)

It’s easier to change the technical elements of a social system than the human ones (enough said).

Maier and Rechtin have an entire appendix at the end of the book on heuristics for system-level architecting. Most of these are intended for software and hardware architecting; however, several have bearing for HR 3200 and the general effort for health care reform.

Plan to throw one away; you will anyway.

This comes from Fred Brooks’ classic work, The Mythical Man-Month, and appears to be highly relevant to what’s going on right now in Congress, where both conservative Democrats and Republicans are suggesting that the best approach right now would be to start over again.

In architecting a new [software] program all the serious mistakes are made in the first day.

As someone who has been dealing since 1995 with failed or troubled IT projects, I find that this is the maxim I keep coming back to. I think that the Obama Administration and the Democratic leadership in Congress badly miscalculated public support for rushing sweeping (and unexamined) health care reform into law given the profound economic problems facing the country (not to mention the massive Federal deficits).

Given a successful organization or system with valid criteria for success, there are some things it cannot do — or at least not do well. Don’t force it!

As noted in Part I, HR 3200 is “kitchen sink” legislation, trying to accomplish a variety of changes that are not necessarily related or dependent. I suspect that Obama and Congress would have been far more successful with a series of small, focused bills that had clear goals and clear limits. The problem with HR 3200 is that by trying to cover so much ground, it merely increases the overall size of the opposition — people with objections to a specific portion of HR 3200 find themselves uniting (directly or indirectly) with those objecting to other portions of HR 3200. By recasting HR 3200 into smaller, well-defined chunks, the opposition to any given chunk becomes smaller as well, increasing that bill’s chances of passage.

Group elements that are strongly related to each other, separate elements that are unrelated.

The shorthard version of this in software design is “high cohesion within a module, loose coupling between modules”. This is another argument for breaking up health care reform into smaller, well-defined and clearly-focused chunks.

If you don’t understand the existing system, you can’t be sure you’re re-architecting a better one.

And, I might add, if you don’t understand the proposed system, you can’t be sure it’s a better one. It is unclear that most of the members of Congress who are pushing HR 3200 understand either the current US health care system or HR 3200 itself (and all its implications).

I could include many more maxims here, but you are better off getting Maier and Rechtin’s book and reading it for yourself.

Part III will suggest a different approach to health care legislation using good practices from systems development and software engineering.

[Part II is now up.]

On the occasions where I have reviewed the actual text of major legislation, I have been struck by the parallels between legislation and software, particularly in terms of the pitfalls and issues with architecture, design, implementation, testing, and deployment. Some of the tradeoffs are even the same, such as trading off the risk of “analysis paralysis” (never moving beyond the research and analysis phase) and the risks of unintended consequences from rushing ill-formed software into production. Yet another similarity is that both software and legislation tend to leverage off of, interact with, call upon, extend, and/or replace existing software and legislation.  Finally, the more complex a given system or piece of legislation is, the less likely that it will achieve the original intent.

But there are some critical differences that make legislation design both harder and higher-risk than systems design. (More after the jump.)

Software vs. Legislation

First, software is designed for a target or reference system; you can in theory predict or constrain its behavior, and its behavior is largely repeatable.

Legislation, by contrast, is executed by humans, with wide latitude for interpretation and implementation, as well as misunderstandings, disagreements on meaning, and on-the-fly modifications.

Second, software typically has several layers of independent (non-human) syntactic, semantic, and integration checking that it of necessity goes through before deployment (though plenty of defects can and do  slip through).

Legislation, by contrast, is written in a natural (human) language, with all its gaps, faults, and ambiguities, and with nothing to force error checking in syntax, semantics, and integration; there’s no way of “compiling“, “linking” and doing a test run of the legislation in a limited environment before it becomes the (largely irrevocable) law of the land.

Third, because of the previous two factors, two or more software engineers can typically reach professional agreement on what a given section of source code will do; if they continue to disagree, there are standard tools and methods by which they can objectively demonstrate how the software will behave, either exactly or within general limits.

By contrast, and due to the corresponding factors with legislation, two or more people (legislators, executives, judges, and citizens) can interpret a given section of legislation quite differently, and each may well have a defensible position, due to the potentially wide latitude of and arena for interpretation.

Some Design Flaws of HR 3200

This all comes to mind as I have been reviewing HR 3200, aka the House bill on health care reform. While I am neither a legislator nor a lawyer (though I have worked closely with lawyers for a decade), I am a professional software architect/engineer, and a professional writer, who has worked in the IT field for 35 years. From that point of view, I believe HR 3200 will exhibit profound problems and unintended (or unclaimed) consequences if passed. Here are some of reasons why.

To begin with, HR 3200 suffers from all the problems listed above with legislation. It is written in English, and complex, obscure, jargon-laden English at that. Many of the sections are imprecise and/or incomplete, leaving large amounts of interpretation and implementation to unelected humans. Many of the objections to HR 3200 come from this very problem, including the concern that the ambiguity is deliberate and intended to open doors to politically unpalatable consequences.

HR 3200 is also massive and very complex — over 1000 pages in printed form, with hundreds of sections. For its sheer length alone, it is difficult to understand and interpret, but (as indicated below) there are other factors that make overall comprehension nearly impossible. It also makes after-the-fact revocation or even modification extremely difficult.

Much of HR 3200 makes piecemeal modifications to existing legislation, often with little explanation as to intent and consequences. So, for example:

SEC. 1148. DURABLE MEDICAL EQUIPMENT PROGRAM IMPROVEMENTS.

…

(c) Treatment of Current Accreditation Applications- Section 1834(a)(20)(F) of such Act (42 U.S.C. 1395m(a)(20)(F)) is amended–

(1) in clause (i)–

(A) by striking ‘clause (ii)’ and inserting ‘clauses (ii) and (iii)’;

(B) by striking ‘and’ at the end;

(2) by striking the period at the end of clause (ii)(II) and by inserting ‘; and’; and

(3) by adding at the end the following:

‘(iii) the requirement for accreditation described in clause (i) shall not apply for purposes of supplying diabetic testing supplies, canes, and crutches in the case of a pharmacy that is enrolled under section 1866(j) as a supplier of durable medical equipment, prosthetics, orthotics, and supplies.

Any supplier that has submitted an application for accreditation before August 1, 2009, shall be deemed as meeting applicable standards and accreditation requirement under this subparagraph until such time as the independent accreditation organization takes action on the supplier’s application.’

This happens repeatedly throughout HR 3200; in fact, one entire portion (Division A, Title IV) is labeled “AMENDMENTS TO INTERNAL REVENUE CODE OF 1986″. This makes it difficult — beyond the ambiguities of the language itself — to determine just what is being modified and what the potential implications are.

HR 3200 also suffers in places from what a software engineer would call “spaghetti coding“. In other words, a given section within HR 3200 (and there appear to be hundreds of them; numbers go from 100 up through 2531 and appear in numeric order, but there are many gaps along the way) will reference several other sections elsewhere in HR 3200, both above and below. Furthermore, it often requires careful reading going back pages to see whether a reference to a given section is to a section within HR 3200 itself or a section in existing legislation (such as the Internal Revenue Service code).

HR 3200 also comes across as similar to a “kitchen sink” application, that is, a single piece of legislation that attempts to do far too much.  I will finish Part I with the table of contents for HR 3200 to give you a sense of all that it is attempting to do. Note that these divisions, titles, and subtitles could have been broken up into individual legislation.

Finally, HR 3200 embodies what is commonly known in software engineering as a “big bang” approach to systems development. In other words, HR 3200 attempts a massive and ill-understood (and/or ill-specified) modification to the nation’s health care system (roughly 1/6th of the economy) in one fell swoop. As such, it really represents the worst excesses of the waterfall development lifecycle, with deployment being hard or impossible to reverse.

Part II discusses software architecture and development maxims, laws, and rules of thumb that appear to have application to creating legislation as well.

Part III will suggest a different approach to health care legislation using good practices from systems development and software engineering.

H.R.3200 – America’s Affordable Health Choices Act of 2009 (table of contents)

DIVISION A–AFFORDABLE HEALTH CARE CHOICES

TITLE I–PROTECTIONS AND STANDARDS FOR QUALIFIED HEALTH BENEFITS PLANS

Subtitle A–General Standards

Subtitle B–Standards Guaranteeing Access to Affordable Coverage

Subtitle C–Standards Guaranteeing Access to Essential Benefits

Subtitle D–Additional Consumer Protections

Subtitle E–Governance

Subtitle F–Relation to Other Requirements; Miscellaneous

Subtitle G–Early Investments

TITLE II–HEALTH INSURANCE EXCHANGE AND RELATED PROVISIONS

Subtitle A–Health Insurance Exchange

Subtitle B–Public Health Insurance Option

Subtitle C–Individual Affordability Credits

TITLE III–SHARED RESPONSIBILITY

Subtitle A–Individual Responsibility

Subtitle B–Employer Responsibility

TITLE IV–AMENDMENTS TO INTERNAL REVENUE CODE OF 1986

Subtitle A–Shared Responsibility

Subtitle B–Credit for Small Business Employee Health Coverage Expenses

Subtitle C–Disclosures To Carry Out Health Insurance Exchange Subsidies

Subtitle D–Other Revenue Provisions

DIVISION B–MEDICARE AND MEDICAID IMPROVEMENTS

TITLE I–IMPROVING HEALTH CARE VALUE

Subtitle A–Provisions Related to Medicare Part A

Subtitle B–Provisions Related to Part B

Subtitle C–Provisions Related to Medicare Parts A and B

Subtitle D–Medicare Advantage Reforms

Subtitle E–Improvements to Medicare Part D

Subtitle F–Medicare Rural Access Protections

TITLE II–MEDICARE BENEFICIARY IMPROVEMENTS

Subtitle A–Improving and Simplifying Financial Assistance for Low Income Medicare Beneficiaries

Subtitle B–Reducing Health Disparities

Subtitle C–Miscellaneous Improvements

TITLE III–PROMOTING PRIMARY CARE, MENTAL HEALTH SERVICES, AND COORDINATED CARE

TITLE IV–QUALITY

Subtitle A–Comparative Effectiveness Research

Subtitle B–Nursing Home Transparency

Subtitle C–Quality Measurements

Subtitle D–Physician Payments Sunshine Provision

Subtitle E–Public Reporting on Health Care-Associated Infections

TITLE V–MEDICARE GRADUATE MEDICAL EDUCATION

TITLE VI–PROGRAM INTEGRITY

Subtitle A–Increased Funding To Fight Waste, Fraud, and Abuse

Subtitle B–Enhanced Penalties for Fraud and Abuse

Subtitle C–Enhanced Program and Provider Protections

Subtitle D–Access to Information Needed To Prevent Fraud, Waste, and Abuse

TITLE VII–MEDICAID AND CHIP

Subtitle A–Medicaid and Health Reform

Subtitle B–Prevention

Subtitle C–Access

Subtitle D–Coverage

Subtitle E–Financing

Subtitle F–Waste, Fraud, and Abuse

Subtitle G–Puerto Rico and the Territories

Subtitle H–Miscellaneous

TITLE VIII–REVENUE-RELATED PROVISIONS

TITLE IX–MISCELLANEOUS PROVISIONS

DIVISION C–PUBLIC HEALTH AND WORKFORCE DEVELOPMENT

TITLE I–COMMUNITY HEALTH CENTERS

TITLE II–WORKFORCE

Subtitle A–Primary Care Workforce

Subtitle B–Nursing Workforce

Subtitle C–Public Health Workforce

Subtitle D–Adapting Workforce to Evolving Health System Needs

TITLE III–PREVENTION AND WELLNESS

TITLE IV–QUALITY AND SURVEILLANCE

TITLE V–OTHER PROVISIONS

Subtitle A–Drug Discount for Rural and Other Hospitals

Subtitle B–School-Based Health Clinics

Subtitle C–National Medical Device Registry

Subtitle D–Grants for Comprehensive Programs To Provide Education to Nurses and Create a Pipeline to Nursing

Subtitle E–States Failing To Adhere to Certain Employment Obligations

]]> http://brucefwebster.com/2009/09/07/hr-3200-from-a-systems-design-perspective-part-i/feed/ 7 http://brucefwebster.com/2008/11/20/five-books-every-it-manager-should-readright-now/ http://brucefwebster.com/2008/11/20/five-books-every-it-manager-should-readright-now/#comments Fri, 21 Nov 2008 03:01:39 +0000 bfwebster http://brucefwebster.com/?p=77 My latest Baseline column  is up, and it talks about why you should read these five books now, if you haven’t already.

And if you have read them, you should probably re-read them.  ..bruce..

Here’s my abstract of the talk:

INSIDE-OUT: Organizations too often treat software reliability as an ‘after the fact’ consideration, performing testing as a last step and then constraining it due to schedule and financial pressures. Webster will present a simple “inside-out” software lifecycle model where all software development activing (not just coding) takes place within a framework covering a broad spectrum of quality-related activities.

I’ll post the presentation slides here after the talk. ..bruce w..

Dr. Ashton was an outstanding instructor. I took this class in 1974, and all our programming was done on an IBM 360/65 mainframe in 360/Assembly, on punched cards no less. Yet Dr. Ashton thoroughly inculcated in us the concepts of structured programming, having us (for example) come up with half a dozen different ways of implementing a WHILE loop in assembler.

At the same time, he did his best to teach us the realities of software development and engineering. One of his comments in class seemed so axiomatic that I immediately thought of it as “Ashton’s Law” and have quoted it as such ever since:

Ashton’s Law: Whenever someone tries to do something for you, they usually end up doing it to you.  — Dr. Alan Ashton, 1974

I have long lost track of how many times over the past 34 years I’ve thought of Ashton’s Law or have spoken it out loud to others. Dr. Ashton made the comment in the context of operating systems, software tools, and applications; all we have to do is look at something like the User Account Protection in MS Vista to see that Ashton’s Law is all too much alive and well.

But there is a deeper issue here in software design and implementation, which I’m also sure Dr. Ashton had in mind. When two software components have to interact, there is usually an explicit and documented (by the source code itself, if by no other means) interface for one component (the ‘called’ component, for lack of a better term) that the other (the ‘calling’ component) makes use of.

In designing functions or services of the called component, we often end up as perpetrators of Ashton’s Law. That is, in our desire to do something for those who will invoke these functions or services, we end up doing something to them.  We make decisions about preconditions, calculations, and results that may seem quite reasonable to us, and that we may have the best intentions for; still, our decisions may lead to others cursing our names (or at least our functions).

There are always tradeoffs in software development among such desirable features as reliability, performance, memory footprint, calling/service standards, security, simplicity, and so on. Our priorities — even our requirements — may differ from those who will use our component’s functions and services, and so a certain amount of friction is perhaps unavoidable. What we can do is spell out all our assumptions and restrictions, and the reasons behind them, so that those using our functions and services know what they’re getting (or not getting).

However, there may exist also what is variously termed a deep, hidden or undocumented interface for the called component. This deep interface deals with invisible and often unknown requirements, side effects, and consequences of a given function or service provided by the called component. For example, the function or service may leave some aspect of the software system or the host computer system in a different state (e.g., changes to internal data structures, changes to files or databases, launching or killing of processes, and so on) without making that fact explicitly clear in any documentation or the API itself. You may feel there is a good reason for this, but it may also interfere with the calling component’s on-going functions.

As with much of what I post here, this may seem quite obvious to you, but trust me — obvious or not, it’s incredibly common and quite frustrating for those having to use the offending software. As software engineers creating software components, systems, and applications to be used by others, we need to constantly keep Ashton’s Law in mind and ensure that we are truly developing useful functionality for others instead simply making their lives more difficult.  ..bruce..

Second, IT engineers by nature tend to be optimists, as reflected in the common acronym SMOP: “simple matter of programming.” Even when an IT engineer doesn’t have a given subsystem completed, he tends to carry with him the notion that he whip everything into shape with a few extra late nights and weekends of effort, even though he may actually face weeks (or more) of work. (NOTE: my use of male pronouns is deliberate; it is almost always male IT engineers who have this unreasonable optimism. Female IT engineers in my experience are generally far more conservative and realistic, almost to a fault, which is why I prefer them. I just wish they weren’t so hard to find.)

Now, a post over at the Wall Street Journal cites what I think is a more controversial (and harder to support) claim — by a female VP of Engineering — that female programmers tend to write clearer and better-documented code than male programmers:

Emma McGrattan, the senior vice-president of engineering for computer-database company Ingres–and one of Silicon Valley’s highest-ranking female programmers–insists that men and women write code differently. Women are more touchy-feely and considerate of those who will use the code later, she says. They’ll intersperse their code–those strings of instructions that result in nifty applications and programs–with helpful comments and directions, explaining why they wrote the lines the way they did and exactly how they did it.

The code becomes a type of “roadmap” for others who might want to alter it or add to it later, says McGrattan, a native of Ireland who has been with Ingres since 1992.

Men, on the other hand, have no such pretenses. Often, “they try to show how clever they are by writing very cryptic code,” she tells the Business Technology Blog. “They try to obfuscate things in the code,” and don’t leave clear directions for people using it later. McGrattan boasts that 70% to 80% of the time, she can look at a chunk of computer code and tell if it was written by a man or a woman.

I’m not sure that I could make the same claim without some serious research into a broad range of coding samples. But the article is worth reading for the comments that follow it, which (as you can imagine) are quite intense.  ..bruce..

And so, from hour to hour, we ripe and ripe,
And then, from hour to hour, we rot and rot;
And thereby hangs a tale.

– William Shakespeare, As You Like It, Act II, Scene vii.

I have observed a pattern (or anti-pattern) in IT engineering that looks something like this:

Let’s call this the arc of engineering, since that’s more compact and elegant than the strangely shaped curve with what appears to be a single inflection point of engineering. “Arc” in any case conveys the essential sense: that system quality (however you wish to define that) rises over time to a peak value and then starts to decline.

This came to mind because I recently put a new hard drive in my laptop and have been going through the tedious drill of reinstalling all the various applications and utilities that I am used to using. With some software, I want — or have to use — the latest versions. So, for example, my versions of Firefox, Thunderbird, Quicken, and so on are all current, not to mention my anti-virus/firewall software.

For others, however, I either don’t need — or specifically don’t want — the newest versions. The laptop’s restore disks installed Windows XP, for which I’m grateful; I have no plans whatsoever to upgrade to Vista. Similarly, I installed MS Office 2003; I don’t own and don’t plan to buy Office 2007. As far as I can tell, Microsoft pretty much hit the top of its engineering arc for both Windows and Office a few years ago. In fact, the early to mid-2000s may well represent the peak of quality, features, and user acceptability for these product lines. Given the on-going end-user unhappiness with Vista and Office 2007 — and the corporate resistance to upgrades to either — Microsoft’s user base may never again be as happy with these product lines as they were a few years ago.

This is not an anti-Microsoft screed, because I have observed a similar effect in other companies and product lines. For example, I was a very early Macintosh adopter and champion, though I also was quite critical of its initial limitations. Apple continued to make improvements and changes to the Macintosh product line and finally reached what I considered at the time to be a close-to-perfect Mac: the Macintosh IIcx. It was solid, reliable, and very stable. Apple followed up with the Macintosh IIci, which used the same form factor, but offered several improvements over the IIcx, though there were some software stability problems due to the rewritten ROM and related hardware changes.

And then came the Macintosh IIfx, which was supposed to be even better — but the IIfx soon became well known for its hardware problems, in particular its tendency to blow out power supplies. Other Mac II models — the IIvi, IIsi, and IIvx — each had their own problems and limitations. From there, Apple descended into the product-line chaos known as Quadra/Centris/Performa, a marketing disaster from which Apple would not recover completely until Steve Jobs came back to Apple and introduced the iMac.

In this same time period, Apple had to deal with the fact that its operating system — System 7, introduced in 1991 — was under-featured, underwhelming, and built on aging technology. System 7 would lose its graphical UI edge with the introduction in 1992 of MS Windows 3.1 and would pretty much lose the operating systems wars completely a few years later with the introduction of MS Windows 95. It would not be until Apple’s introduction in 2002 of the Jaguar (10.2) version of Mac OS X — a heavily updated and retooled version of the NeXTstep OS — that Apple would regain technical and UI leadership in operating systems.

This same pattern can appear in internal IT projects. The system under development goes into production and continues to improve over time. Then at some point, the system’s overall quality (functionality, performance, reliability) starts to decline and maintenance costs begin to go up. At some point the organization decides to replace or re-engineer the system, sometimes successfully, sometimes not.

So, why does this happen? I believe several factors are or can be involved. (To simply the discussion, I’ll use “system” to mean the software, IT system, product, or product line under discussion; likewise, I’ll use “developer” to mean the company and/or IT development group responsible for creating, enhancing, and maintaining the system.)

The developer loses conceptual control of the system. In other words, no person or group of people understands the overall architecture, design, and operating constraints of the system. Thus the developer can no longer ensure that on-going changes to the system are consistent and compatible with existing aspects of the system or with other changes being made. This in turn can come about for several reasons:

• The system grows too large and complex. Sheer code size is not necessarily a problem (though it doesn’t help); on the other hand, the degree of fragmentation and interdependencies can be. (See this Vista post-mortem for some examples.)
• Those who really understand the system have left or moved on to other projects. The best and brightest IT engineers — particularly architects and designers — usually prefer to work on new systems. That means once this system has shipped (or gone into production), the people who created the system in the first place and who understand it best may well want to move on to other things, inside or outside of the company. Those who replace them — and those who stay behind — may be quite talented but may not understand all the system’s nuances and requirements. Eventually, they may decide to move on as well. With each generation of turnover, more institutional memory is lost.
• The development organization no long meshes with the system. Conway’s law states that “any piece of software reflects the organizational structure that produced it.” Since the IT group responsible for the system often changes once the system ships or goes into production, the organization of the new group responsible for the system may not match the system’s structure, causing gaps in knowledge and responsibility, plus a tendency — conscious or not — to reshape the system to match the new organization.

Software rot sets in. Software rot is a real term of art within software engineering, referring to the well-known tendency of systems in production to become less reliable over time for a variety of reasons having mostly to do with piecemeal and uncoordinated changes to the system, or growing incompatibilities with external systems.

The enhanced system finally outgrows its original foundation. When the developer seeks to expand or improve the system, it often does so without properly rearchitecting and redesigning the system first. As a result, the original system eventually may be stressed beyond its original intent and begins to suffer from functionality, reliability and/or performance problems. Mike Parker, one of the co-founders of Pages Software Inc. (read the 3rd paragraph), once described this syndrome as “trying to build a castle on the foundation of an outhouse.”

Market or business needs shift beyond the product’s fundamental design. A developer creates and enhances a system to meet a specific set of business/market requirements. However, there often comes a time when new or expanded requirements exceed the capabilities of the system’s original architecture and design. This can lead to the “outgrowing the foundation” situation described above, with the resulting problems. Or it can lead to a true re-engineering or replacement effort, which may well lead to the final two problems below.

The developer begins to add “blue sky”/”kitchen sink” enhancements. Once the “door is open” on changes to the system — or features for the new/re-engineered system — the requests pour in from all directions. Without rigorous control over these enhancements, the development effort can quickly bloat in every aspect: architecture, design, code size, schedule, staff, and defects.

Backward compatibility is maintained at all costs. Supporting backwards compatibility in a system can be a bit like trying to drive a car while pedaling a wheel at the same time. Apple abandoned its old arc of engineering and created a new one under Mac OS X by first forcing binary backwards compatibility into a separate mode (the “Classic” environment) and then dropping the Classic environment altogether for Intel-based Macs running Mac OS X. Microsoft — which has struggled with the backwards compatibility issue for almost its entire OS life — has announced that for “Windows 7″, it will in effect follow Apple’s lead by not supporting binary (and to a certain extent source) backwards compatibility in Windows 7 but instead using virtualization technology (switching to a backwards-compatible OS) for older application.

I’m sure there are other reasons I haven’t listed here, but it does show why the arc of engineering exists. Comments? ..bruce..

Humanity has been developing information technology for half a century. That experience has taught us this unpleasant truth: virtually every information technology project above a certain size or complexity is significantly late and over budget or fails altogether; those that don’t fail are often riddled with defects and difficult to enhance. Fred Brooks explored many of the root causes over twenty years ago in The Mythical Man-Month, a classic book that could be regarded as the “Bible” of information technology because it is universally known, often quoted, occasionally read, and rarely heeded. Most publications and books on IT since then have debated, discussed, and deplored these same problems. And they are with us still. Their causes stem not from technology but from human frailties. Indeed, when asked why so many IT projects go wrong in spite of all we know, one could simply cite the seven deadly sins: avarice, sloth, envy, gluttony, wrath, lust, and pride. It is as good an answer as any and more accurate than most.

– Testimony given by Bruce F. Webster before the Subcommittee on Government Management, Information, and Technology, United States House of Representatives, June 22, 1998.

Many, if not most, information technology (IT) failures result from human factors, not technical ones. In my dual roles as an IT consultant to large organizations since 1995 and as an expert witness in IT systems failure litigation since 1999, I have spent thousands of hours reviewing documents, interviewing developers, managers, and executives, analyzing software lifecycle deliverables, reading deposition transcripts, pouring through error tracking logs and even reviewing source code. The goal in all this is to figure out (depending upon my role) why a project is or was troubled. In some cases, it may be because of reliance upon software or hardware that was unexpectedly flawed or insufficiently capable, or due to some unexpected external circumstances. But in the majority of cases, it really does come down to human factors — and usually one or more of the seven deadly sins cited above.

I now recite those sins in a different order and with slightly different wording — pride, envy, greed, lust, anger, gluttony, sloth — because that way they form the handy mnemonic “PEG LAGS” (with apologies to all Margarets and Peggys reading this). That’s the order I’ll address them in subsequent posts. ..bruce..