Second, I have chronicled here my problems with two HP systems — a desktop and a laptop — that I each purchased new, with Windows 7 Home Premium (64-bit) pre-installed (and in each case upgraded to Win7Pro).  With both systems, I have had chronic problems with blue screens of death (BSODs), particularly when using a wireless connection, though sometimes even when being used with a wired connection. As I wrote last June, what was most telling was my acquisition of a third Win7HP system (also upgraded to Win7Pro 64-bit), a Gateway desktop system. Unlike the two HP systems, where I had lots of BSODs (sometimes a few dozen in a single day), I had never had a BSOD on the Gateway system. The HP BSOD problems would come and go in waves, but they have nevertheless persisted on the two HP systems. I did do a ‘factory restore’ on the laptop at Christmas (as I mentioned I might last June), but that didn’t cure the BSOD problem. As a result, I have been seriously considering buying a new (non-HP) laptop, and ultimately replacing my HP desktop as well. In the meantime, the Gateway kept chugging along flawlessly.

Until a few weeks ago. You see, I took the Gateway system with me out of town on a business trip because I needed to be able to use certain files and applications that were stored on it. The client site didn’t have a wired LAN connection for me, but they did have a wireless access point I could use. So I went out and purchased a wireless USB adapter for the Gateway, got it working after a bit (that’s another story), and then promptly got my first-ever BSOD on the Gateway. And then another. And then another. Like my prior BSODs on the HP systems, I’d get BSODs without even being logged into an account. In the meantime, back at my hotel, I was getting multiple BSODs on my HP laptop whether I was connected wired or wirelessly. I had to switch to my Sprint 3G/4G wireless dongle (yet another story) to avoid those problems.

This forced me to look around a bit more on the net for those with similar problems. I found them: several forums where people described the same symptoms. What was the common theme? Having Zone Alarm Security Suite installed, which I have used for years, and which I had installed on all three machines.

Facepalm.

So I removed Zone Alarm from all three system, replacing it with Microsoft Security Essentials. In the ten (10) days since doing that, I haven’t had a single BSOD on any system, even though I was averaging a bit more than 1 BSOD/day on my laptop alone (47 from January 3rd to February 4th).

I leave you, the reader, to draw your own conclusions.  ..bruce..

Since last November, I have bought three new, out-of-the-box systems preinstalled with Windows 7 Home Premium 64-bit  (and upgraded to Windows 7 Professional 64-bit at the end of May):

• an HP Pavilion e9237c desktop (quad-core 64-bit  processor, 8 GB RAM, 1 TB hard drive) [purchased early November 2009]
• an HP Pavilion dv7 laptop (dual-core 64-bit processor, 6 GB RAM, 600 GB hard drive) [purchased early March 2010]
• a Gateway SX2802 desktop (quad-core 64-bit processor, 4 GB RAM, 750 GB hard drive) [purchased mid-April 2010]

As I’ve written previously here, I’ve had on-going problems with ‘blue screes of death’ (BSODs) with the HP desktop system, which may or may not be tied to the graphics system. The c:\windows\minidump folder only keeps the last 50 BSOD dumps; a quick review shows that I’ve had 50 BSODs since January 20th of this year. However, given the fact that I was on the road for most of the period from late March through early July, and so not using my desktop very much, that number may be artificially low. For example, I have been home continuously since July 6th; in that time, I have had eleven (11) BSODs.

What I haven’t written about here are the extreme BSOD problems I’ve been having with the HP laptop since buying it earlier this year (early March). I have had literally hundreds of BSODs in that time and would have had more if I had not discovered at least one major factor: a wireless connection. If I have the wireless adapter enabled, not only do I get frequent BSODs while trying to work, I will get BSODs when the computer simply has the login screen up.

Case in point: ten days ago, on July 11th, I had my laptop powered up and the wireless adapter enabled.  My laptop then experienced twenty-seven (27) BSODs from 8:27 am to 2:54 pm, almost all of which happened without me being at the computer or even being logged in (from the windows\minidump directory):

[date]               [time]         [file size]  [file name]

07/11/2010 08:27 AM 296,376 071110-20638-01.dmp

07/11/2010 08:33 AM 296,376 071110-18408-01.dmp

07/11/2010 08:54 AM 296,376 071110-17877-01.dmp

07/11/2010 09:00 AM 296,376 071110-17908-01.dmp

07/11/2010 09:09 AM 296,376 071110-17737-01.dmp

07/11/2010 09:19 AM 296,376 071110-16832-01.dmp

07/11/2010 09:21 AM 296,376 071110-17066-01.dmp

07/11/2010 09:42 AM 296,376 071110-18501-01.dmp

07/11/2010 09:51 AM 296,376 071110-16317-01.dmp

07/11/2010 09:54 AM 296,376 071110-16208-01.dmp

07/11/2010 10:09 AM 296,376 071110-16848-01.dmp

07/11/2010 10:12 AM 296,376 071110-16489-01.dmp

07/11/2010 10:21 AM 296,376 071110-16333-01.dmp

07/11/2010 10:24 AM 296,376 071110-17144-01.dmp

07/11/2010 10:30 AM 296,376 071110-16692-01.dmp

07/11/2010 10:36 AM 296,376 071110-17019-01.dmp

07/11/2010 10:45 AM 296,376 071110-16551-01.dmp

07/11/2010 10:54 AM 296,376 071110-16442-01.dmp

07/11/2010 11:00 AM 296,376 071110-16863-02.dmp

07/11/2010 11:24 AM 296,376 071110-17035-02.dmp

07/11/2010 12:42 PM 296,376 071110-17004-01.dmp

07/11/2010 12:45 PM 296,376 071110-18408-02.dmp

07/11/2010 12:58 PM 296,376 071110-17565-01.dmp

07/11/2010 01:00 PM 296,376 071110-17877-02.dmp

07/11/2010 01:09 PM 296,376 071110-17596-01.dmp

07/11/2010 02:48 PM 296,376 071110-18002-01.dmp

07/11/2010 02:54 PM 296,376 071110-16957-01.dmp

[UPDATE] I ran the WhoCrashed app on the latest 50 crashes on my laptop; it reported that it could only analyze 40 out of the 50. But those forty were, as far as I could tell, identical except for the ‘Bugcheck Code’ line; for example:

On Sun 7/11/2010 3:21:44 PM your computer crashed
This was likely caused by the following module: ntoskrnl.exe
Bugcheck code: 0x7F (0×8, 0×80050033, 0x6F8, 0xFFFFF80003299E58)
Error: UNEXPECTED_KERNEL_MODE_TRAP
Dump file: C:\Windows\Minidump71110-17066-01.dmp
file path: C:\Windows\system32\ntoskrnl.exe
product: Microsoft® Windows® Operating System
company: Microsoft Corporation
description: NT Kernel & System
The crash took place in a standard Microsoft module. Your system configuration may be incorrect, possibly the culprit is in another driver on your system which cannot be identified at this time.

[END UPDATE]

I do experience occasional BSODs on the laptop when I have the wireless adapter turned off and the laptop hooked up via an ethernet cable to a network, but they are much less frequent. But here’s the really interesting part. Because of my heavy travels, I bought a Sprint ‘net dongle (model U301) and the accompanying Sprint data service plan. I could be wrong, but I don’t think I’ve had a single BSOD when connected to the Sprint network via the Sprint dongle.

In all this, my presumption has been that the problem has been with Microsoft, and it may still be. But back in mid-April, while home for a few days, I bought the Gateway desktop system because I needed a separate system that I could pull off the network to do some significant hard disk decryption and encryption (typically running 5-6 days straight each time). So the system has been powered up and running since mid-April, and has had during that time two 6-day periods of round-the-clock CPU- and I/O-intensive processing. (It is, in fact, the system I am using to write this post.)

Total number of BSODs on the Gateway system since I bought it three months ago: zero (0). Nada. None whatsoever. The c:\windows\minidump folder is empty.

The Gateway system has at least two key differences from the HP systems. First, it has no wireless adapter; the HP desktop has an Atheros 802.11 a/b/g/n Dualband wireless module, while the HP laptop has an Intel Centrino Advanced-N 6200 AGN. Second, the Gateway uses an Intel G45/G43 integrated graphics chipset, while the HP desktop uses an ATI Radeon HD 4650 and the HP laptop uses an NVIDIA GeForce G105M.

The HP desktop is stable “enough” that I have no plans of replacing or returning it at the moment. As for the HP laptop, I am considering doing a clean factory restore  (and, if that fails to work, taking it back in; it’s under extended warranty), but it’s going to be a pain, particularly given all the software (including the Win7Pro upgrade) that I have installed on it.

Note, by the way, that I’ve seen no difference in the BSOD rate on either HP system since upgrading each to Windows 7 Professional at the end of May. Note also that ‘Automatically download and install Windows updates’ is enabled for all three systems, so they are all in sync with the latest Windows upgrades and with each other.

Anyone else had similar problems with newer HP systems running Windows 7?  ..bruce..

As noted below, last fall I bought an HP Pavillion desktop (quad-core 64-bit  processor, 8 GB ram, 1 TB hd) running Windows 7 (Home Premium 64-bit). It’s got an ATI Radeon 4650 with a chunk of dedicated RAM (512 MB, I think) and a 27″ HD display (1920 x 1080). It all worked more or less fine until a few months ago, when I suddenly started getting random Blue Screens of Death (BSODs), usually while playing a game or watching  a video for a while. Note that some of the games involved weren’t terribly graphic intensive (such as Civ4). Likewise, I had it happen while viewing a YouTube video embedded in a blog. Also note that I had no problems with the exact same games and video applications prior to that point.

As mentioned, I thought that the video card or something else might have gone flaky on me. I do have  a 3-year service warranty on the system, but didn’t want to drag my desktop in to where I bought it. I’ve been swamped for the last month, so I’ve largely ignored the problem.

However, it came up today while I was installing software for a new printer. There was an animated disk icon rotating on the screen, and — kablooey! BSOD! This time I went digging on Google, and I wasn’t happy with what I found. Poking around on several different forums, I found folks describing a variety of problems with ATI Radeon video cards and Windows 7 updates, including in Microsoft forums.

In particularly, I found a couple of posts describing pretty much the same situation I face, namely that things worked well for a while, but random BSODs associated with game and other video-related applications started up after a Windows 7 update sometime a few months ago.

Thank you, Microsoft. I was strongly considering converting over to all Apple hardware and software (with XP dual boot), but was seduced both by the low prices of Wintel hardware and the improvements of Windows 7 over Vista. Between this and my earlier problem, I’m starting to regret it.  ..bruce..

I was in the process of copying some files from elsewhere on my internal LAN onto my desktop’s internal drive — I believe I had two sets of copying going on, one from another laptop, another from a network hard drive. I was also downloading a new version of a particular software program that was already installed. The instructions for that program suggested uninstalling the previous version before installing the new one, so I started doing that as well. (Hey — Win7 is supposed to be a multi-tasking OS, right?)

The de-installation threw up an alert box stating that such-and-such a file could not be deleted due to permission issues. I clicked the ‘OK’ button (my only choice) and waited for the de-installation to complete or terminate.  It did neither, but pretty much hung without making any progress on its glowing green progress bar. I finally grew tired of waiting, brought up Task Manager, and killed the de-installation.

At that point, my desktop redrew itself, except that now almost all of the icons were missing. Uh-oh.

I keep most of my work in folders (not shortcuts, actual folders) on my desktop. Likewise, I tend to park files on the desktop until I decide where to to file them. l I immediately looked at the contents of “C:\Users\<username>\Desktop” and saw that it was empty. Completely empty.

I used a couple of different file recovery packages in an attempt to recover what had vanished. No luck. They found no deleted folders or files within the Desktop folder, and my efforts at doing more extensive scans ran for a few days straight — while still being less than 50% done — before I gave up. (Since getting the desktop, I had used it to collect vast file sets, move them onto external hard drives, then delete those sets. There were likely at least a few million deleted files with traces all over the 1 TB hard drive.)

Before you wag your fingers at me, I did have an external 1.5 TB hard drive to which I was running weekly backups. However, I had just a week or so before the glitch (and a few days after the last backup) moved my desktop system from the main floor to my office downstairs and had not re-attached that external hard drive. This means, of course, that I was able to recover the deleted folders, but the versions were about 10 days old. I didn’t lose a lot, but then, I’m not entirely sure what all I did lose.

Yesterday, I discovered that the same glitch had apparently emptied out the  “MyMusic” folder, wiping out my iTunes library. I restored much of that from my laptop (which had an older version of my iTunes library), but lost everything that I had purchased from the iTunes store and downloaded onto my desktop system.

I’m still not sure what happened. I did a fair amount of searching on the web for similar problems and found that other people had had the same thing happen to them. One posting suggested that my user profile may have been corrupted, but my efforts to fix things that way didn’t help.

I bought a second external hard drive for backups and plan to run them more frequently. I also rearranged things so that I no longer have actual folders within Desktop but only shortcuts to folders elsewhere.

Just a word to the wise.  ..bruce..

This naturally intrigued me, since for the last 15 years, I have been writing, consulting, lecturing, and testifying about troubled and failed IT projects. While there are indeed tremendous financial losses due to late and failed IT projects, the figures Sessions gives seem much too large to me, and so I decided to do this critique of his analysis.

Sessions is good enough to provide the basis of his estimates and calculations, including footnotes. But that’s where some of the problems start. For example,  on page 3, Sessions cites (his footnote ’02′) to the US Budget, Fiscal Year 2009, Analytical Perspective (PDF), p. 169, for information on “at-risk” or failed IT projects, specifically:

• “According to the 2009 U.S. Budget [02], the failure rate is increasing at the rate of around 15% per year. If this trend continues, within another five years or so a total IT meltdown may be unavoidable.” (p. 3)
• “According to the 2009 U.S. Budget [02], 66% of all Federal IT dollars are invested in projects that are ‘at risk’. I assume this number is representative of the rest of the world.” (p. 3, in “Calculating the Cost of IT Failure” box)
• A large number of these ['at risk' projects] will eventually fail. I assume the failure of an ‘at risk’ project is between 50% and 80%. For this analysis, I’ll use the average: 65%.”

These three statements run into immediate problems. First, and relatively minor, Sessions gets his page number wrong: he’s citing “page 169″ of the Analytical Perspective document, but there is no discussion whatsoever on page 169 of that document about IT projects. However, page 157 of that document (which happens to be page 169 of the PDF document) does start a section titled “INTEGRATING SERVICES WITH INFORMATION TECHNOLOGY”, so I presume that Sessions made the simple mistake of using the PDF page count rather than the document’s actual page numbering.

Even so, serious problems remain with Sessions’ citations and analysis.

Page 157 of the Analytical Perspective document does not say what Sessions claimed in the two comments above. I have not been able to figure out where Sessions gets his figure for “the failure rate increasing around 15% per year” from the cited US Budget Analytical Perspective document, much less his conclusion that “if this trend continues, within another five years or so a total IT meltdown may be unavoidable.” As far as I can tell, the Analytical Perspective document does not talk about failed IT projects at all, much less the increase in failure rates.

Furthermore, the phrase “the failure rate increasing around 15% per year” is itself ambiguous and may not be that significant. To start with an arbitrary number, assume that 100 projects “fail” in a given year. If “the failure rate [is] increasing around 15% per year”, then that means that 115 projects would fail the next year, and 132 projects would fail the year after that. But unless we know both the actual number of failed IT projects and the total number of IT projects in that same year, Sessions’ figure tells us nothing. If there’s only 150 IT projects total, then the 15% failure rate increase becomes very significant; if there’s 1000 IT projects total, then we’re many years away from Sessions’ threatened “meltdown”.

Sessions also ignores or confuses the failure rate for new projects vs. the systems already deployed. In other words, the failure rate for new systems development says very little about the continued functionality of existing, deployed systems now in use. While there are occasions (most notably Y2k, now a decade behind us) where existing IT systems just won’t function or function properly if they aren’t fixed or replaced, by and large both governments and private concerns have gotten along remarkably well for years or even decades with antiquated systems

As for Sessions’ second statement, there is a table on page 158 that may represent the basis for it:

As can be seen in the FY 2009 column, 66% (535 out of 810) of the FY 2009 “Major IT Investments” are projects that are “Not Well Planned and Managed”. Note that this table does not (as Sessions infers) indicate Federal dollars but rather actual projects; that is, in FY 2009, there are 810 projects listed as “Major IT investments”, of which 535 are designated as “Not Well Planned and Managed”. The previous page appears to indicate that these projects represent $27 billion, which is roughly 38% of the proposed Federal IT budget — not a great figure, but still almost half of the 66% that Sessions claims.

What’s more, supplementary data (PDF) for the FY 2009 Analytical Perspective makes it clear that the US Government’s designation of such projects — which puts them on a “Management Watch List” (WML) — has reduced the risk of such projects during each fiscal year:

Note that in FY 2007 and 2008, the number of IT projects designated as “Not Well Planned and Managed” shrunk significantly during the year (from Q1 to Q4) without a proportional shrinkage of the overall number of major IT projects. In other word, it appears that the government’s efforts to remove such projects from the “Not Well Planned and Managed” category is relatively successful. And the actual US IT budget dollars at risk at the end of each of those fiscal years ($4.2 billion for FY 07, $8.6 billion for FY 08)  is a much smaller percentage (6.5% and 13%, respectively) of the Federal IT budget for each of those years ($64.2 billion for FY 07 (XLS), $66.4 billion for FY 08 (XLS)).

Sessions then states that “I assume this number [66% of all Federal IT dollars being at risk] is representative of the rest of the world.” There are numerous problems with this assumption, starting with the fact that the 66% figure is wrong; in fact, the actual “at risk” (his term, not the US Government’s) percentage of the IT budget at the end of FY 07 and FY 08 were, as noted above, 6.5% and 13%, respectively.

Sessions’ error here is significant, since he goes on in several places (cf. page 4) to cite his use of the % of the total IT budget as being significant, when he’s not talking about the total IT budget at all.

Furthermore, it is unclear whether his phrase “the rest of the world” means all other national governments, or all other entities doing IT project development. It seems to be the latter, though it’s hard to tell from his statements. On the other hand, I have spent years consulting with corporations on troubled projects, and I can tell you that they do not have 66% of their IT budgets devoted to “at risk” projects. In fact, the majority of corporate IT budgets are devoted to maintenance of existing systems, not new and risky projects (cf. here, here, here, and here, as simple examples).

As noted, Sessions then assumes that the failure rate for “at risk” IT projects is 65%, which means that (as he says) “I am calculating that 43% (.65 x .66) of the total IT budget” is devoted to failed projects. At this point, his figures become nonsensical, as they are derived both from misreadings and lack of complete information about the Federal IT budget and projects. To wit:

• The 535 “not well planned and managed” IT projects in the US FY 09 budget only represent 38% of the total IT budget, not 66% as Sessions mistakenly states.
• In the two previous years (FY 07 and FY 08), the number of IT projects labeled as “not well planned and managed” dropped during the course of each year (see the 2nd table above). In FY 07, it dropped from 263 projects in Q1 to just 84 in Q4, which means that 69% were moved off of the “not well planned and managed” list during the year. Likewise, in FY 08, it dropped from 346 projects in Q1 to 134 projects in Q4, a drop of 61%. This directly contradicts Sessions’ assumption of a 65% failure rate for projects in the “not well planned and managed” category.
• The FY ’09 Analytical Perspective says nothing about actual failed projects, as far as I can tell.

Sessions then goes on to make further out-of-his-hat assumptions regarding “direct and indirect costs”. He cites an example of the IRS (an agency long troubled by IT woes) and notes a lost opportunity based on fraudulent tax returns due to the system not being operational. He projects a loss over two years ($1.788 billion), compares it to the cost of the failed modernization ($185 million over a ten-year period), and calculates an indirect costs ratio of 9.6 to 1. He then decides — with no other documentation or analysis whatsoever — that the universal ratio of indirect to direct costs for a failed IT project ranges from 5:1 to 10:1, and uses the “average” of 7.5:1.

There are so many problems here that I scarce know where to start. For starters, the term “average” assumes an even distribution of ratios from 5:1 to 10:1 and does not recognize any ratios lower than 5:1. I’ve seen many failed projects that had much lower ratios of “indirect” to “direct” costs, since the firm simply continued to operate using the existing systems, and the “lost opportunity” for not having the new system in place was relatively small.

More importantly, the IRS gets to collect taxes from the entire US: $2.5 trillion in tax collections each year. Using the IRS as a baseline makes little sense for most other government agencies, and even less sense for most corporations and non-government organizations (NGOs), because most IT systems in most organizations (government or private) do not have the ability to generate such magnitudes of revenue, period.

Indeed, there is a long-standing controversy within IT management circles as to whether a new computer system can be relied upon to provide any significant return on investment (ROI), or whether it exists merely to “keep up with the competition”.

Sessions concludes his section on calculations thusly (p. 5, emphasis his):

Of course, these calculations are estimates. I recommend you don’t get overly focused on the exact amounts. I could be off by ten or twenty percent in either directions. The real point is not the exact numbers, but the magnitude of the numbers and the fact that the numbers are getting worse.

Unfortunately, Sessions is fundamentally wrong in his numerical analysis, and his numbers are off by far more than “ten or twenty percent”. For the Federal Government alone, they are off by almost  a full order of magnitude (10x), due to his critical errors both on the percentage of the Federal IT ’09 budget “at risk” (it’s 38%, not 66%) and the number of “at risk” projects that fail (he says 65%; the US government numbers for FY 07 and 08 show that only 35% of the projects — representing just 6.5% to 13% percent of the Federal IT budget — were still “at risk” at the end of each fiscal year, and it gives no figures that I can find for actual failed IT projects).

Furthermore, his projection of the (erroneous) 66%-of-IT-budget-at-risk figure on the rest of the world is just wrong, especially in corporations and business (which spend vastly more on IT than the US government). In those organizations, maintenance costs dominates, and the percentage of the IT budget devoted to new projects tends to be small (20% or less), with an even smaller fraction of that representing “at risk” projects.

I may comment more on Sessions’ paper, but my conclusion here is that his estimate of $500 billion/month in lost direct and indirect costs due to IT systems failure just does not hold up, in my opinion.  ..bruce..

Here in Part II, I’ll talk about some of the well-established maxims and heuristics of complex systems development, and how they apply to legislation in general and to HR 3200 in particular. (More after the jump.)

Gall

As far as I can tell, John Gall — in his out-of-print book Systemantics (1976)– was the first to observe in print that

A complex system that works is found to have invariably evolved from a simple system that worked. (p. 80, 1978 paperback edition).

Immediately after, he observes that:

A complex system designed from scratch never works and cannot be made to work. You have to started over,  beginning with a working simple system.

My co-blogger (over at ASIP) Bruce Henderson puts this another way:

Start out stupid, and work up from there.

There is large room for differing arguments here as to just where HR 3200 fits in, for several reasons.

First, HR 3200 isn’t “designed from scratch.” As noted in Part I, many sections of HR 3200 are modifying various existing laws and regulations, such as the Internal Revenue Code, the Public Health Service Act, Employee Retirement Income Security Act, the Social Security Act, and the United States Code.

However, leveraging upon and modifying several existing systems is not the same as building a “simple system that works” and evolving it into a complex system that works. I can create a large, complex piece of software that calls upon and even modifies existing systems and libraries — but that doesn’t necessarily mean I’m evolving something from a “small, simple system that works”. This is especially true when I’m pulling together from several disjoint or unrelated systems (such as those listed above).

Second, legislation is more robust than software, for exactly the differences outlined in part I, namely that legislation is executed by people rather than machines and operating systems. If I create an ill-formed piece of software, there’s a good chance it won’t even compile (or interpret); if it does, then it may run into linking or integration errors; and if it gets past those, it may crash, lock up, or behave bizarrely upon execution.

If, however, I create an ill-formed piece of legislation, it can be (and often is!) be put into practice, with various human either officially or unofficially working around the defects to make it “work”. Of course, that ‘deployment’ of the legislation may end up drifting or even veering sharply from the stated or actual intent of the legislation. (In a way, this is reminiscent of the early PL/1 compilers that would, upon encountering a syntax error, make a best guess as to what you might have meant to write and compile that instead.)

Courts can shift this ‘deployment’ in both directions. They may “find” meaning or functionality in the law never contemplated or even explicitly disavowed by those who crafted and voted for the legislation, or they may prohibit some portion of explicit functionality due to conflicts with the Constitution, prior judicial rulings, or simply their own judgment.  As noted in Part I, judges don’t always agree with one another, either, so whether a given piece of legislation (or a subportion thereof) is upheld, modified, or rejected entirely depends upon which courts or individual judges end up reviewing it.

Third, there are serious and compelling arguments as to how well the current government health care programs (such as Medicare and the VA hospital system) work, not to mention the government systems modified and relied upon by HR 3200 (such as the IRS and Social Security). While you may argue with Gall’s maxims above, I know of no serious systems designer who will state that it is possible to build a large, complex system that works from complex systems that work poorly, if at all. The quality of your original and leveraged systems provides an upper bound on the quality of your final system. To believe otherwise is to succumb to wishful thinking.

Maier and Rechtin

In The Art of Systems Architecting by Mark W. Maier and Eberhardt Rechtin (2002), the authors take a cross-discipline approach to systems architecting, including talking specifically about social systems in Chapter 5. The following passage from that chapter is of particular relevance to the overall purpose of HR 3200 (all emphasis in the original):

The first insight, which might be called the four whos, asks four questions that need to be answered as a self-consistent set if the system is to succeed economically; namely, who benefits? who pays? who provides? and, as appropriate, who loses?

The political arguments raging over HR 3200 are exactly over those four questions. In fact, Maier and Rechtin themselves foresaw those arguments, since they go on to use health care as an example:

Example: serious debates over the nature of their public health services are underway in many countries, triggered in large part by the technological advances of the last few decades. These advances have made it possible for humanity to live longer and in better health, but the investments in those gains are considerable. The answer to the four whos are at the crux of the debate. Who benefits — everyone equally at all levels of health? Who pays — regardless of personal health or based on need and ability to pay? Who provides — and determines cost to the user? Who loses — anyone out of work or above some risk level, and who determines who loses?

The problem with HR 3200 and with the arguments put forth to date on its behalf is that they have not systematically and credibly addressed those four questions. In fact, those arguing in support of HR 3200 and health care reform in general have often given contradictory answers to those four questions, undermining their own credibility, given ammo to their opposition, and (justifiably) undermining public support for HR 3200.

Along those lines, the authors also note that in architecting social systems, you face not just the constraints of normal system design — risk, performance, schedule, and cost — but two more: perception vs. facts. They go on to say:

Social systems have generated a painful design heuristic: it’s not the facts, it’s the perception that counts. Some real-world examples: . . .

• One of the reasons that health insurance is so expensive is that health care is perceived by employees as nearly “free” because almost all its costs are paid for either by the the employee’s company or the government. The facts are that the costs are either passed on to the consumer, subtracted from wages and salary, taken as a business deduction against taxes, or all of the above. There is no free lunch.

Again, with great relevance to the current debate over HR 3200 and the whole approach of the House over health care reform, the authors state:

Like it or not, the architect must understand that perceptions can be just as real as facts, just as important in defining the system architecture, and just as critical in determining success. As one heuristic states: the phrase, ‘I hate it’, is direction. There have even been times when, in retrospect, perceptions were “truer” than facts which changed with observer, circumstance, technology, and better direction. . . . In the end, it is a matter of achieving a balance of perceived values. The architect’s task is to search out that area of common agreement that can result in a desirable, feasible system.

Maier and Rechtin end Chapter 5 with some heuristics they consider specific to social systems. Several are those already cited above, but here are a few additional ones (my comments are in brackets):

Success is the eye of the beholder [i.e., the US public] (not the architect [i.e., Congress]).

Don’t assume that the original statement of the problem [e.g., "45 million uninsured"] is necessarily the best, or even the right one. (Most customers would agree.)

In social systems, how you do something may be more important than what you do. (A sometimes bitter lesson for technologists [and Congress] to learn.)

It’s easier to change the technical elements of a social system than the human ones (enough said).

Maier and Rechtin have an entire appendix at the end of the book on heuristics for system-level architecting. Most of these are intended for software and hardware architecting; however, several have bearing for HR 3200 and the general effort for health care reform.

Plan to throw one away; you will anyway.

This comes from Fred Brooks’ classic work, The Mythical Man-Month, and appears to be highly relevant to what’s going on right now in Congress, where both conservative Democrats and Republicans are suggesting that the best approach right now would be to start over again.

In architecting a new [software] program all the serious mistakes are made in the first day.

As someone who has been dealing since 1995 with failed or troubled IT projects, I find that this is the maxim I keep coming back to. I think that the Obama Administration and the Democratic leadership in Congress badly miscalculated public support for rushing sweeping (and unexamined) health care reform into law given the profound economic problems facing the country (not to mention the massive Federal deficits).

Given a successful organization or system with valid criteria for success, there are some things it cannot do — or at least not do well. Don’t force it!

As noted in Part I, HR 3200 is “kitchen sink” legislation, trying to accomplish a variety of changes that are not necessarily related or dependent. I suspect that Obama and Congress would have been far more successful with a series of small, focused bills that had clear goals and clear limits. The problem with HR 3200 is that by trying to cover so much ground, it merely increases the overall size of the opposition — people with objections to a specific portion of HR 3200 find themselves uniting (directly or indirectly) with those objecting to other portions of HR 3200. By recasting HR 3200 into smaller, well-defined chunks, the opposition to any given chunk becomes smaller as well, increasing that bill’s chances of passage.

Group elements that are strongly related to each other, separate elements that are unrelated.

The shorthard version of this in software design is “high cohesion within a module, loose coupling between modules”. This is another argument for breaking up health care reform into smaller, well-defined and clearly-focused chunks.

If you don’t understand the existing system, you can’t be sure you’re re-architecting a better one.

And, I might add, if you don’t understand the proposed system, you can’t be sure it’s a better one. It is unclear that most of the members of Congress who are pushing HR 3200 understand either the current US health care system or HR 3200 itself (and all its implications).

I could include many more maxims here, but you are better off getting Maier and Rechtin’s book and reading it for yourself.

Part III will suggest a different approach to health care legislation using good practices from systems development and software engineering.

[Part II is now up.]

On the occasions where I have reviewed the actual text of major legislation, I have been struck by the parallels between legislation and software, particularly in terms of the pitfalls and issues with architecture, design, implementation, testing, and deployment. Some of the tradeoffs are even the same, such as trading off the risk of “analysis paralysis” (never moving beyond the research and analysis phase) and the risks of unintended consequences from rushing ill-formed software into production. Yet another similarity is that both software and legislation tend to leverage off of, interact with, call upon, extend, and/or replace existing software and legislation.  Finally, the more complex a given system or piece of legislation is, the less likely that it will achieve the original intent.

But there are some critical differences that make legislation design both harder and higher-risk than systems design. (More after the jump.)

Software vs. Legislation

First, software is designed for a target or reference system; you can in theory predict or constrain its behavior, and its behavior is largely repeatable.

Legislation, by contrast, is executed by humans, with wide latitude for interpretation and implementation, as well as misunderstandings, disagreements on meaning, and on-the-fly modifications.

Second, software typically has several layers of independent (non-human) syntactic, semantic, and integration checking that it of necessity goes through before deployment (though plenty of defects can and do  slip through).

Legislation, by contrast, is written in a natural (human) language, with all its gaps, faults, and ambiguities, and with nothing to force error checking in syntax, semantics, and integration; there’s no way of “compiling“, “linking” and doing a test run of the legislation in a limited environment before it becomes the (largely irrevocable) law of the land.

Third, because of the previous two factors, two or more software engineers can typically reach professional agreement on what a given section of source code will do; if they continue to disagree, there are standard tools and methods by which they can objectively demonstrate how the software will behave, either exactly or within general limits.

By contrast, and due to the corresponding factors with legislation, two or more people (legislators, executives, judges, and citizens) can interpret a given section of legislation quite differently, and each may well have a defensible position, due to the potentially wide latitude of and arena for interpretation.

Some Design Flaws of HR 3200

This all comes to mind as I have been reviewing HR 3200, aka the House bill on health care reform. While I am neither a legislator nor a lawyer (though I have worked closely with lawyers for a decade), I am a professional software architect/engineer, and a professional writer, who has worked in the IT field for 35 years. From that point of view, I believe HR 3200 will exhibit profound problems and unintended (or unclaimed) consequences if passed. Here are some of reasons why.

To begin with, HR 3200 suffers from all the problems listed above with legislation. It is written in English, and complex, obscure, jargon-laden English at that. Many of the sections are imprecise and/or incomplete, leaving large amounts of interpretation and implementation to unelected humans. Many of the objections to HR 3200 come from this very problem, including the concern that the ambiguity is deliberate and intended to open doors to politically unpalatable consequences.

HR 3200 is also massive and very complex — over 1000 pages in printed form, with hundreds of sections. For its sheer length alone, it is difficult to understand and interpret, but (as indicated below) there are other factors that make overall comprehension nearly impossible. It also makes after-the-fact revocation or even modification extremely difficult.

Much of HR 3200 makes piecemeal modifications to existing legislation, often with little explanation as to intent and consequences. So, for example:

SEC. 1148. DURABLE MEDICAL EQUIPMENT PROGRAM IMPROVEMENTS.

…

(c) Treatment of Current Accreditation Applications- Section 1834(a)(20)(F) of such Act (42 U.S.C. 1395m(a)(20)(F)) is amended–

(1) in clause (i)–

(A) by striking ‘clause (ii)’ and inserting ‘clauses (ii) and (iii)’;

(B) by striking ‘and’ at the end;

(2) by striking the period at the end of clause (ii)(II) and by inserting ‘; and’; and

(3) by adding at the end the following:

‘(iii) the requirement for accreditation described in clause (i) shall not apply for purposes of supplying diabetic testing supplies, canes, and crutches in the case of a pharmacy that is enrolled under section 1866(j) as a supplier of durable medical equipment, prosthetics, orthotics, and supplies.

Any supplier that has submitted an application for accreditation before August 1, 2009, shall be deemed as meeting applicable standards and accreditation requirement under this subparagraph until such time as the independent accreditation organization takes action on the supplier’s application.’

This happens repeatedly throughout HR 3200; in fact, one entire portion (Division A, Title IV) is labeled “AMENDMENTS TO INTERNAL REVENUE CODE OF 1986″. This makes it difficult — beyond the ambiguities of the language itself — to determine just what is being modified and what the potential implications are.

HR 3200 also suffers in places from what a software engineer would call “spaghetti coding“. In other words, a given section within HR 3200 (and there appear to be hundreds of them; numbers go from 100 up through 2531 and appear in numeric order, but there are many gaps along the way) will reference several other sections elsewhere in HR 3200, both above and below. Furthermore, it often requires careful reading going back pages to see whether a reference to a given section is to a section within HR 3200 itself or a section in existing legislation (such as the Internal Revenue Service code).

HR 3200 also comes across as similar to a “kitchen sink” application, that is, a single piece of legislation that attempts to do far too much.  I will finish Part I with the table of contents for HR 3200 to give you a sense of all that it is attempting to do. Note that these divisions, titles, and subtitles could have been broken up into individual legislation.

Finally, HR 3200 embodies what is commonly known in software engineering as a “big bang” approach to systems development. In other words, HR 3200 attempts a massive and ill-understood (and/or ill-specified) modification to the nation’s health care system (roughly 1/6th of the economy) in one fell swoop. As such, it really represents the worst excesses of the waterfall development lifecycle, with deployment being hard or impossible to reverse.

Part II discusses software architecture and development maxims, laws, and rules of thumb that appear to have application to creating legislation as well.

Part III will suggest a different approach to health care legislation using good practices from systems development and software engineering.

H.R.3200 – America’s Affordable Health Choices Act of 2009 (table of contents)

DIVISION A–AFFORDABLE HEALTH CARE CHOICES

TITLE I–PROTECTIONS AND STANDARDS FOR QUALIFIED HEALTH BENEFITS PLANS

Subtitle A–General Standards

Subtitle B–Standards Guaranteeing Access to Affordable Coverage

Subtitle C–Standards Guaranteeing Access to Essential Benefits

Subtitle D–Additional Consumer Protections

Subtitle E–Governance

Subtitle F–Relation to Other Requirements; Miscellaneous

Subtitle G–Early Investments

TITLE II–HEALTH INSURANCE EXCHANGE AND RELATED PROVISIONS

Subtitle A–Health Insurance Exchange

Subtitle B–Public Health Insurance Option

Subtitle C–Individual Affordability Credits

TITLE III–SHARED RESPONSIBILITY

Subtitle A–Individual Responsibility

Subtitle B–Employer Responsibility

TITLE IV–AMENDMENTS TO INTERNAL REVENUE CODE OF 1986

Subtitle A–Shared Responsibility

Subtitle B–Credit for Small Business Employee Health Coverage Expenses

Subtitle C–Disclosures To Carry Out Health Insurance Exchange Subsidies

Subtitle D–Other Revenue Provisions

DIVISION B–MEDICARE AND MEDICAID IMPROVEMENTS

TITLE I–IMPROVING HEALTH CARE VALUE

Subtitle A–Provisions Related to Medicare Part A

Subtitle B–Provisions Related to Part B

Subtitle C–Provisions Related to Medicare Parts A and B

Subtitle D–Medicare Advantage Reforms

Subtitle E–Improvements to Medicare Part D

Subtitle F–Medicare Rural Access Protections

TITLE II–MEDICARE BENEFICIARY IMPROVEMENTS

Subtitle A–Improving and Simplifying Financial Assistance for Low Income Medicare Beneficiaries

Subtitle B–Reducing Health Disparities

Subtitle C–Miscellaneous Improvements

TITLE III–PROMOTING PRIMARY CARE, MENTAL HEALTH SERVICES, AND COORDINATED CARE

TITLE IV–QUALITY

Subtitle A–Comparative Effectiveness Research

Subtitle B–Nursing Home Transparency

Subtitle C–Quality Measurements

Subtitle D–Physician Payments Sunshine Provision

Subtitle E–Public Reporting on Health Care-Associated Infections

TITLE V–MEDICARE GRADUATE MEDICAL EDUCATION

TITLE VI–PROGRAM INTEGRITY

Subtitle A–Increased Funding To Fight Waste, Fraud, and Abuse

Subtitle B–Enhanced Penalties for Fraud and Abuse

Subtitle C–Enhanced Program and Provider Protections

Subtitle D–Access to Information Needed To Prevent Fraud, Waste, and Abuse

TITLE VII–MEDICAID AND CHIP

Subtitle A–Medicaid and Health Reform

Subtitle B–Prevention

Subtitle C–Access

Subtitle D–Coverage

Subtitle E–Financing

Subtitle F–Waste, Fraud, and Abuse

Subtitle G–Puerto Rico and the Territories

Subtitle H–Miscellaneous

TITLE VIII–REVENUE-RELATED PROVISIONS

TITLE IX–MISCELLANEOUS PROVISIONS

DIVISION C–PUBLIC HEALTH AND WORKFORCE DEVELOPMENT

TITLE I–COMMUNITY HEALTH CENTERS

TITLE II–WORKFORCE

Subtitle A–Primary Care Workforce

Subtitle B–Nursing Workforce

Subtitle C–Public Health Workforce

Subtitle D–Adapting Workforce to Evolving Health System Needs

TITLE III–PREVENTION AND WELLNESS

TITLE IV–QUALITY AND SURVEILLANCE

TITLE V–OTHER PROVISIONS

Subtitle A–Drug Discount for Rural and Other Hospitals

Subtitle B–School-Based Health Clinics

Subtitle C–National Medical Device Registry

Subtitle D–Grants for Comprehensive Programs To Provide Education to Nurses and Create a Pipeline to Nursing

Subtitle E–States Failing To Adhere to Certain Employment Obligations

]]> http://brucefwebster.com/2009/09/07/hr-3200-from-a-systems-design-perspective-part-i/feed/ 7 http://brucefwebster.com/2009/03/04/fireflies-conveyor-belts-and-landfills/ http://brucefwebster.com/2009/03/04/fireflies-conveyor-belts-and-landfills/#comments Wed, 04 Mar 2009 17:35:18 +0000 bfwebster http://brucefwebster.com/?p=117 My newest Baseline column is up, and in it, I talk about technology lifecycles that can cause you grief:

Each technology is on its own product lifecycle, which may or may not match with your organization’s business and development lifecycles. In particular, there are certain cycle mismatch patterns that commonly occur in organizations looking to adopt new technologies. I’ve labeled four such mismatch patterns: firefly, underdone, conveyer belt, and landfill. Each is worth examining.

Go read the whole thing.  ..bruce..

In the meantime, I have two new Baseline columns out that deal with risk management in IT project. I give both a bad example and a good example, each drawn from my professional experience. Comments, as always, are welcome here or at Baseline.  ..bruce..