Back in July 2010, I bought an Acer Aspire easyStore Home Server running Windows Home Server 2003, primarily two solve two problems: (1) run nightly backups of all my Wintel PCs; (2) provide a common media file repository for myself and my wife (who runs Macs). After a few bumps, it worked just fine and has been working fine since.

Except that a few months ago — which at this point may mean something like this past summer or even late spring, given how time flies — I noticed that I could no longer run the WHS Console app from any of my PCs (all running Win7 Pro). Any attempt to do so resulted in an error box with the message:

This computer cannot connect to your home server. Check your network connection and make sure your home server is powered on. If your home server has recently restarted, try again in a few minutes.

Here’s what’s strange: the server itself still shows up in the Network panel, I can still access all the shares, and I can still log onto the server via the Remote Desktop Connection application (and, by the way, run the Console app there, on the server itself). I can ping the server in a console window via its name and its (fixed) IP address. And the automated backups keep happening (though more on this below).

At roughly the same time — and it may have been exactly the same time — I began to get certificate warnings when I used RDC to log into the server. This is a well-known problem with a kludgy work-around (configure RDC to ignore certificate warnings). That problem appears to be associated with Service Pack 1 of Windows 7, and I suspect the Console problem showed up at the same time as well.

However, to date, I have not found a workaround to the Console problem. What’s more, I’m starting to have backup problems. In particular, one desktop system stopped backing up. In an effort to fix that, I uninstalled the WHS Connector Software and attempted to re-install it. No go — I get this error during installation, after entering the server’s admin password:

This operation cannot be completed at this time.

Please try again later. If the problem persists, please contact Product Support.

This appears to be a well-known, if intermittent, problem, but one for which a variety of suggested-but-not-necessarily-effective solutions have been offered. I started to tackle the problem again this morning, trying some of the solutions, and found some other oddities:

• Even though I can see my server (which we’ll call foo-bar for now) in the Network panel, can access it, can ping it (by name or by IP), and can remotely log into it, I cannot bring it up in a browser, e.g., http://foo-bar or even http://<fixed IP address>. That last one is particularly surprising.
• Yesterday, I brought down and then brought back up my entire home-wide network (modem, router, access point, lots of computers and other network-accessing devices). I noticed that the Acer server box took forever (probably 15+ minutes) to reboot itself and reappear on the network.
• I’ve been getting flaky behavior from WHS itself over the past few months when I remotely log in (which I don’t do that often). For example, when I logged in today, I got three standard Windows “unexpected error – send info to Microsoft?” boxes, all associated with the backup utility.
• I did bring up the Computer Management panel on the server, thinking that the inability to browse to the server may depending on something running here. I did find that IIS Admin Services was not running and would not start, even though its two dependencies (RPC and Security Accounts) were, in fact, running.

I suspect that what I may actually need to do is reinstall WHS 2003, but that means that I will have to (for safety’s sake) back up the (non-automated) shares. Not a bad thing to do anyway, but not how I envisioned spending my Thanksgiving weekend.

What struck me while researching on-line is how many people had posts that said, in effect, “I did this and it fixed everything”, followed by posts saying, “I tried that and it still doesn’t work”. Similarly, not everyone running the same systems gets the same problems. It goes to the heart of the underlying complexity, uncertainty, and — to a certain extent — unknowability of the systems upon which we depend.

That said, I welcome suggestions.   ..bruce..

The first key step was buying a Windows Home Server box last summer. It took me a while to get all the kinks out (read my review at the link), but since then it was worked pretty much trouble-free, 24/7. Not only do my various computers get backed up each night, but it’s become the standard iTunes repository for both my wife and me, and we use it to share (legally) media that we purchase separately on iTunes.

The next step was upgrading our internet access. We live in a semi-rural area outside of Denver, and when we moved here 6 years ago, our only internet access options were a dedicated T1 line (our next-door neighbor had one), satellite access, or a wide-area wireless (with dedicated dish pointing to an access point about 6 miles away). We went with the last option, which was given us roughly DSL-grade access (1.5Mb). However, about six months ago, we started getting flyers from Qwest claiming that high-speed phone line access was finally available in our area. We signed up and found ourselves with access speeds running typically from 4 to 12 Mbits/sec. Not much compared to some of the very high speed (>100 Mbits) access available in some cities now, but a major step up from what we’ve been living with.

The third step was getting a new HD large-screen TV this past week — in this case, as an astonishingly generous gift from our youngest daughter and her boyfriend. We bought our existing Sony 42″ plasma TV some 8+ years ago, and I won’t tell you what we paid for it then. Not only was it getting long in the tooth, with the display losing contrast, but the TV itself only had one (1) set of component inputs; the other four were all composite/S-video, and there were no HDMI ports at all. The new TV (a 55″ Sony LCD) has a much sharper and clearer picture as well as up-to-date ports (4 HDMI ports, an optical audio out, etc.).

The fourth step was buying an Apple TV unit today. This not only gives us Netflix streaming, it recognizes and lets us access that iTunes media share on our WHS box. [UPDATE: But wait! Not really! See below.] After getting it hooked up, I sat down and watched an episode of “Castle” that I had downloaded from iTunes (and moved onto the server) but had not yet watched.I then listened to some of the music on the server, then went out and connected to one of what appear to be at least a few hundred streaming radio stations from the ‘net. Oh, and I watched a few minutes of both a TV show and a movie from Netflix (we are long-time Netflix subscribers).

To quote the great Steverino, it all just works.  And it works beautifully, too — the video and sound quality is outstanding.

I’ll note in here that a few days ago I bought a Sony Blu-Ray player with ‘net connectivity. It actually sees more of the computer on our home network, but it does not recognize or play iTunes media. If it did, I probably wouldn’t have bought the Apple TV unit.

As per the headline, this is almost enough to persuade me to drop our DirecTV subscription. Almost, but not enough. I love college football and especially love watching it in high-def. I also watch (or at least have on in the background) a lot of local and national news. If and when I have streaming options for those that I like, then DirecTV will likely go away; it’s just not worth the cost.

I give it a year or so.  ..bruce..

UPDATE: Thought I was going crazy there for a while. After seeing the stuff on my WHS box, I went back later and found I could only see stuff on my Win7 laptop (which is where I keep all my ‘active’  iTunes media and where I sync my iPhone and my iPad). I thought maybe I misread what I was seeing, but then I disabled and then re-enabled iTunes sharing on my WHS box, then brought up iTunes on my Win7 laptop. The WHS iTunes library showed up in my laptop’s iTunes app — and now all the media files on my WHS box are showing up as well. But it will only play the files on my laptop. Any attempt to select, say, a TV show episode that’s only on the WHS box results in an error message.

So, I’ve got a couple of choices: install iTunes on my WHS box (which is not as straightforward as I would like) or move all the iTunes media on the WHS box onto Sandra’s MacPro (where her iTunes library resides and which, fortunately, has a couple of terabytes of free internal disk space). Since the Apple TV readily sees her iTunes library, and since her MacPro (unlike my laptop) is almost always on, that may be the easiest solution. Stay tuned.

I poked around on-line a bit, found some other similar complaints, but found no clear solution or even explanation as to what was causing the problem. Since the iPad doesn’t expose its file system, I had no direct way (via Apple software) of examining it. So I went out and bought a 3rd-party application (the regular kind, not an iPad app) named Phone Disk, which — like several other similar apps — allows the iPad to be mounted as if it were an external hard drive. It worked as advertised, and some browsing via the Finder found that the iTunes Music folder held 40 GB of files — which was more or less the sum of the Music, Video, and Other segments as displayed in iTunes. None of the other folders on the iPad could account for that extra 10 GB, so it was pretty clear that the 10 GB of “Other” was hiding amongst the music and video files.

Next step was to unsync all music and video from her iPad. That done, the 9.9 GB of “Other” still showed up in the iTunes display. Checked the iPad directly with Phone Disk and, yes, there were still 10 GB of files in the iTunes/Music folder. I started opening the individual subfolders in Music and found a random assortment of audio and video files: songs, TV shows, movies (including at least one expired rental), even though the iPod and Video apps both claimed there was no music nor video on the iPad at all. There were never more than 3 or 4 files in a given subfolder, and many subfolders had no files as all.

I deleted all these files manually (via Phone Disk/Finder), then re-synced the iPad. Boom! (as Steve would say): that 9.9 GB of “Other” was now a tiny sliver measured in megabytes.

I’m not sure what was causing these file to be left behind, unrecognized by the iPad and Video apps; I did note that a large number (though not all) of the unrecognized files all had the same date (Nov 8, 2010). On the other hand, one of the “left behind” files was dated just this morning (a TV show episode that my wife had downloaded and synced); curiously, in the iTunes library on her MacPro, that TV show episode appeared twice. In any event, it’s clear that there are occasions in which media files are considered by the on-board iPod and Video apps to be wiped from the iPad but nevertheless remain on the iPad file system, taking up space.

Having looked at my wife’s iPad, I synced mine as well, and I see that I have nearly 2 GB of “Other” on mine. I plan to use Phone Disk (I bought a family license) and go through the same examination (and possible cleanup) of my iTunes files on the iPad. I’ll report back if I find anything interesting.  ..bruce..

It didn’t work.The adapter showed up as a network connection, but it was disabled and would not let it self be enabled. I went through some of the usual first steps (uninstall, reboot, re-install), but they didn’t work. I searched the net for newer drivers, and found some — went through the uninstall/reboot/install new drivers cycle. Plugged the USB device it. Same problem. There were comments in some of the on-line forums that this particular adapter didn’t work with Windows 7, so I figured that was likely my problem.

The next day, I stopped by the same store and bought a different USB wireless adapter, this one clearly marketed “Compatible with Windows 7!” Went through the installation cycle, and this one didn’t work either. Looked for newer drivers, found them, downloaded and installed them — still no luck. Used Google to search for similar complaints, and I found them, one of which spoke to my situation. The user had Zone Alarm installed on his system. He said that he had found that he could successfully install the network adapter only if he completely uninstalled Zone Alarm — not just turned it off, but actually uninstalled it — then installed the adapter, then re-installed Zone Alarm. I tried the same thing, and it worked. Of course, as I wrote about two days ago, I then started getting BSODs on my Gateway box, which had never happened before; my eventual solution was to uninstall Zone Alarm and replace it with Microsoft Security Essentials. I frankly wonder in retrospect whether that would have solved my problem with re-installing my Sprint U301 device as well.

I’ve written these last three posts to chronicle these solutions to anyone else who might be having these same problems (and, as I noted two days ago, to apologize to HP and Microsoft for putting the original blame on them for my multitude of BSODs). We’ll now move on to some other topics for a while.  ..bruce..

No problem: I got on the ‘net via the hotel long enough to download the latest and greatest SSV package (version 2.50.0094) from the Sprint support site, installed it, and…could not get my U301 to initialize and connect. The device, when plugged in, appeared to install itself with the requisite drivers, but when I launched SSV to connect to Sprint, SSV would see the device, try to initialize and connect, but never succeed. I did all the usual things (uninstall SSV, uninstall the 301 while deleting the drivers, do a clean reinstall of everything, etc.), but nothing worked.

As it turned out, Sprint had a repair center just four blocks where I was staying. Since I was paying the small monthly fee for equipment protection, I just dropped the U301 off, giving an explanation of my problems. Next day, I get a call: your device works just fine. I swing by the Sprint store, pick up the U301, and later in my hotel room try another clean installation. No luck.

Now, I had purchased the U301 in the spring of 2010, while doing extended work at a customer site back East; it had installed and worked just fine out of the box — same laptop, same OS (less whatever Win7 updates had come out), same system utilities. But I noticed that the SSV software I downloaded was dated 1/13/2011, so it was definitely not the same version I had originally installed.

So, on a hunch, I started searching the ‘net for older versions of the SSV software installation package. I found one: version 2.40xx (vs. 2.50xx on the Sprint site). I removed the 2.50 SSV software and the U301 (with its driver), installed the 2.40 SSV software, and plugged the U301 into my laptop.

Everything installed and ran just fine. It continued to run just fine when the 2.40 software upgraded itself to the 2.50 software.

Now, in light of my simultaneous problems with my Gateway box and the subsequent solution I found, I suspect there might have been another solution. But I’ll talk about that in my next post.  ..bruce..

Most of the electronics that Dad dealt with were old-school: vacuum tubes, custom circuit boards, large and discrete components, and so on. He said that whenever a piece of electronic equipment started acting up, his first course of action — if a quick inspection didn’t reveal a core problem — was to sharply hit the equipment on the top or side. As he explained it, heat and motion tended to loosen connections; a sharp rap would often re-seat those components.

I thought of that tonight when I got a text message from my wife that our dual-band 802.11n router suddenly stopped recognizing her laptop. When I called her, she said that one band wouldn’t accept the household password while the other band (on entering the password) would hang and eventually time out. After talking with her for a minute (we’re about 1,000 miles apart right now), I told her to go power down the router, wait about 30 seconds, power it back up, wait a minute, and then try again.

It worked.

I find it interesting that even as the physical hardware has become smaller, cheaper, more integrated and more reliable, it is our software — the virtual and digital connections — that tend to “come loose” over time. When a piece of digital equipment starts acting funny, how often is our first act — if a first inspection doesn’t reveal a core problem — simply to reboot or power-cycle the equipment? And how often does that, indeed, end up fixing the problem? And it’s not just with home equipment and home systems; I’ve seen the same approach applied to mis-functioning “high-availability” systems in large corporate environments.

This says something, I think, about the fundamental complexity (usually high) and quality (often lower than it should be) of the software, systems, and protocols on which we depend, both personally and professionally. We adapt as individuals and organizations to these systems, rather than having the systems adapt to us. And we all poke along together.  ..bruce..

This naturally intrigued me, since for the last 15 years, I have been writing, consulting, lecturing, and testifying about troubled and failed IT projects. While there are indeed tremendous financial losses due to late and failed IT projects, the figures Sessions gives seem much too large to me, and so I decided to do this critique of his analysis.

Sessions is good enough to provide the basis of his estimates and calculations, including footnotes. But that’s where some of the problems start. For example,  on page 3, Sessions cites (his footnote ’02′) to the US Budget, Fiscal Year 2009, Analytical Perspective (PDF), p. 169, for information on “at-risk” or failed IT projects, specifically:

• “According to the 2009 U.S. Budget [02], the failure rate is increasing at the rate of around 15% per year. If this trend continues, within another five years or so a total IT meltdown may be unavoidable.” (p. 3)
• “According to the 2009 U.S. Budget [02], 66% of all Federal IT dollars are invested in projects that are ‘at risk’. I assume this number is representative of the rest of the world.” (p. 3, in “Calculating the Cost of IT Failure” box)
• A large number of these ['at risk' projects] will eventually fail. I assume the failure of an ‘at risk’ project is between 50% and 80%. For this analysis, I’ll use the average: 65%.”

These three statements run into immediate problems. First, and relatively minor, Sessions gets his page number wrong: he’s citing “page 169″ of the Analytical Perspective document, but there is no discussion whatsoever on page 169 of that document about IT projects. However, page 157 of that document (which happens to be page 169 of the PDF document) does start a section titled “INTEGRATING SERVICES WITH INFORMATION TECHNOLOGY”, so I presume that Sessions made the simple mistake of using the PDF page count rather than the document’s actual page numbering.

Even so, serious problems remain with Sessions’ citations and analysis.

Page 157 of the Analytical Perspective document does not say what Sessions claimed in the two comments above. I have not been able to figure out where Sessions gets his figure for “the failure rate increasing around 15% per year” from the cited US Budget Analytical Perspective document, much less his conclusion that “if this trend continues, within another five years or so a total IT meltdown may be unavoidable.” As far as I can tell, the Analytical Perspective document does not talk about failed IT projects at all, much less the increase in failure rates.

Furthermore, the phrase “the failure rate increasing around 15% per year” is itself ambiguous and may not be that significant. To start with an arbitrary number, assume that 100 projects “fail” in a given year. If “the failure rate [is] increasing around 15% per year”, then that means that 115 projects would fail the next year, and 132 projects would fail the year after that. But unless we know both the actual number of failed IT projects and the total number of IT projects in that same year, Sessions’ figure tells us nothing. If there’s only 150 IT projects total, then the 15% failure rate increase becomes very significant; if there’s 1000 IT projects total, then we’re many years away from Sessions’ threatened “meltdown”.

Sessions also ignores or confuses the failure rate for new projects vs. the systems already deployed. In other words, the failure rate for new systems development says very little about the continued functionality of existing, deployed systems now in use. While there are occasions (most notably Y2k, now a decade behind us) where existing IT systems just won’t function or function properly if they aren’t fixed or replaced, by and large both governments and private concerns have gotten along remarkably well for years or even decades with antiquated systems

As for Sessions’ second statement, there is a table on page 158 that may represent the basis for it:

As can be seen in the FY 2009 column, 66% (535 out of 810) of the FY 2009 “Major IT Investments” are projects that are “Not Well Planned and Managed”. Note that this table does not (as Sessions infers) indicate Federal dollars but rather actual projects; that is, in FY 2009, there are 810 projects listed as “Major IT investments”, of which 535 are designated as “Not Well Planned and Managed”. The previous page appears to indicate that these projects represent $27 billion, which is roughly 38% of the proposed Federal IT budget — not a great figure, but still almost half of the 66% that Sessions claims.

What’s more, supplementary data (PDF) for the FY 2009 Analytical Perspective makes it clear that the US Government’s designation of such projects — which puts them on a “Management Watch List” (WML) — has reduced the risk of such projects during each fiscal year:

Note that in FY 2007 and 2008, the number of IT projects designated as “Not Well Planned and Managed” shrunk significantly during the year (from Q1 to Q4) without a proportional shrinkage of the overall number of major IT projects. In other word, it appears that the government’s efforts to remove such projects from the “Not Well Planned and Managed” category is relatively successful. And the actual US IT budget dollars at risk at the end of each of those fiscal years ($4.2 billion for FY 07, $8.6 billion for FY 08)  is a much smaller percentage (6.5% and 13%, respectively) of the Federal IT budget for each of those years ($64.2 billion for FY 07 (XLS), $66.4 billion for FY 08 (XLS)).

Sessions then states that “I assume this number [66% of all Federal IT dollars being at risk] is representative of the rest of the world.” There are numerous problems with this assumption, starting with the fact that the 66% figure is wrong; in fact, the actual “at risk” (his term, not the US Government’s) percentage of the IT budget at the end of FY 07 and FY 08 were, as noted above, 6.5% and 13%, respectively.

Sessions’ error here is significant, since he goes on in several places (cf. page 4) to cite his use of the % of the total IT budget as being significant, when he’s not talking about the total IT budget at all.

Furthermore, it is unclear whether his phrase “the rest of the world” means all other national governments, or all other entities doing IT project development. It seems to be the latter, though it’s hard to tell from his statements. On the other hand, I have spent years consulting with corporations on troubled projects, and I can tell you that they do not have 66% of their IT budgets devoted to “at risk” projects. In fact, the majority of corporate IT budgets are devoted to maintenance of existing systems, not new and risky projects (cf. here, here, here, and here, as simple examples).

As noted, Sessions then assumes that the failure rate for “at risk” IT projects is 65%, which means that (as he says) “I am calculating that 43% (.65 x .66) of the total IT budget” is devoted to failed projects. At this point, his figures become nonsensical, as they are derived both from misreadings and lack of complete information about the Federal IT budget and projects. To wit:

• The 535 “not well planned and managed” IT projects in the US FY 09 budget only represent 38% of the total IT budget, not 66% as Sessions mistakenly states.
• In the two previous years (FY 07 and FY 08), the number of IT projects labeled as “not well planned and managed” dropped during the course of each year (see the 2nd table above). In FY 07, it dropped from 263 projects in Q1 to just 84 in Q4, which means that 69% were moved off of the “not well planned and managed” list during the year. Likewise, in FY 08, it dropped from 346 projects in Q1 to 134 projects in Q4, a drop of 61%. This directly contradicts Sessions’ assumption of a 65% failure rate for projects in the “not well planned and managed” category.
• The FY ’09 Analytical Perspective says nothing about actual failed projects, as far as I can tell.

Sessions then goes on to make further out-of-his-hat assumptions regarding “direct and indirect costs”. He cites an example of the IRS (an agency long troubled by IT woes) and notes a lost opportunity based on fraudulent tax returns due to the system not being operational. He projects a loss over two years ($1.788 billion), compares it to the cost of the failed modernization ($185 million over a ten-year period), and calculates an indirect costs ratio of 9.6 to 1. He then decides — with no other documentation or analysis whatsoever — that the universal ratio of indirect to direct costs for a failed IT project ranges from 5:1 to 10:1, and uses the “average” of 7.5:1.

There are so many problems here that I scarce know where to start. For starters, the term “average” assumes an even distribution of ratios from 5:1 to 10:1 and does not recognize any ratios lower than 5:1. I’ve seen many failed projects that had much lower ratios of “indirect” to “direct” costs, since the firm simply continued to operate using the existing systems, and the “lost opportunity” for not having the new system in place was relatively small.

More importantly, the IRS gets to collect taxes from the entire US: $2.5 trillion in tax collections each year. Using the IRS as a baseline makes little sense for most other government agencies, and even less sense for most corporations and non-government organizations (NGOs), because most IT systems in most organizations (government or private) do not have the ability to generate such magnitudes of revenue, period.

Indeed, there is a long-standing controversy within IT management circles as to whether a new computer system can be relied upon to provide any significant return on investment (ROI), or whether it exists merely to “keep up with the competition”.

Sessions concludes his section on calculations thusly (p. 5, emphasis his):

Of course, these calculations are estimates. I recommend you don’t get overly focused on the exact amounts. I could be off by ten or twenty percent in either directions. The real point is not the exact numbers, but the magnitude of the numbers and the fact that the numbers are getting worse.

Unfortunately, Sessions is fundamentally wrong in his numerical analysis, and his numbers are off by far more than “ten or twenty percent”. For the Federal Government alone, they are off by almost  a full order of magnitude (10x), due to his critical errors both on the percentage of the Federal IT ’09 budget “at risk” (it’s 38%, not 66%) and the number of “at risk” projects that fail (he says 65%; the US government numbers for FY 07 and 08 show that only 35% of the projects — representing just 6.5% to 13% percent of the Federal IT budget — were still “at risk” at the end of each fiscal year, and it gives no figures that I can find for actual failed IT projects).

Furthermore, his projection of the (erroneous) 66%-of-IT-budget-at-risk figure on the rest of the world is just wrong, especially in corporations and business (which spend vastly more on IT than the US government). In those organizations, maintenance costs dominates, and the percentage of the IT budget devoted to new projects tends to be small (20% or less), with an even smaller fraction of that representing “at risk” projects.

I may comment more on Sessions’ paper, but my conclusion here is that his estimate of $500 billion/month in lost direct and indirect costs due to IT systems failure just does not hold up, in my opinion.  ..bruce..

Here in Part II, I’ll talk about some of the well-established maxims and heuristics of complex systems development, and how they apply to legislation in general and to HR 3200 in particular. (More after the jump.)

Gall

As far as I can tell, John Gall — in his out-of-print book Systemantics (1976)– was the first to observe in print that

A complex system that works is found to have invariably evolved from a simple system that worked. (p. 80, 1978 paperback edition).

Immediately after, he observes that:

A complex system designed from scratch never works and cannot be made to work. You have to started over,  beginning with a working simple system.

My co-blogger (over at ASIP) Bruce Henderson puts this another way:

Start out stupid, and work up from there.

There is large room for differing arguments here as to just where HR 3200 fits in, for several reasons.

First, HR 3200 isn’t “designed from scratch.” As noted in Part I, many sections of HR 3200 are modifying various existing laws and regulations, such as the Internal Revenue Code, the Public Health Service Act, Employee Retirement Income Security Act, the Social Security Act, and the United States Code.

However, leveraging upon and modifying several existing systems is not the same as building a “simple system that works” and evolving it into a complex system that works. I can create a large, complex piece of software that calls upon and even modifies existing systems and libraries — but that doesn’t necessarily mean I’m evolving something from a “small, simple system that works”. This is especially true when I’m pulling together from several disjoint or unrelated systems (such as those listed above).

Second, legislation is more robust than software, for exactly the differences outlined in part I, namely that legislation is executed by people rather than machines and operating systems. If I create an ill-formed piece of software, there’s a good chance it won’t even compile (or interpret); if it does, then it may run into linking or integration errors; and if it gets past those, it may crash, lock up, or behave bizarrely upon execution.

If, however, I create an ill-formed piece of legislation, it can be (and often is!) be put into practice, with various human either officially or unofficially working around the defects to make it “work”. Of course, that ‘deployment’ of the legislation may end up drifting or even veering sharply from the stated or actual intent of the legislation. (In a way, this is reminiscent of the early PL/1 compilers that would, upon encountering a syntax error, make a best guess as to what you might have meant to write and compile that instead.)

Courts can shift this ‘deployment’ in both directions. They may “find” meaning or functionality in the law never contemplated or even explicitly disavowed by those who crafted and voted for the legislation, or they may prohibit some portion of explicit functionality due to conflicts with the Constitution, prior judicial rulings, or simply their own judgment.  As noted in Part I, judges don’t always agree with one another, either, so whether a given piece of legislation (or a subportion thereof) is upheld, modified, or rejected entirely depends upon which courts or individual judges end up reviewing it.

Third, there are serious and compelling arguments as to how well the current government health care programs (such as Medicare and the VA hospital system) work, not to mention the government systems modified and relied upon by HR 3200 (such as the IRS and Social Security). While you may argue with Gall’s maxims above, I know of no serious systems designer who will state that it is possible to build a large, complex system that works from complex systems that work poorly, if at all. The quality of your original and leveraged systems provides an upper bound on the quality of your final system. To believe otherwise is to succumb to wishful thinking.

Maier and Rechtin

In The Art of Systems Architecting by Mark W. Maier and Eberhardt Rechtin (2002), the authors take a cross-discipline approach to systems architecting, including talking specifically about social systems in Chapter 5. The following passage from that chapter is of particular relevance to the overall purpose of HR 3200 (all emphasis in the original):

The first insight, which might be called the four whos, asks four questions that need to be answered as a self-consistent set if the system is to succeed economically; namely, who benefits? who pays? who provides? and, as appropriate, who loses?

The political arguments raging over HR 3200 are exactly over those four questions. In fact, Maier and Rechtin themselves foresaw those arguments, since they go on to use health care as an example:

Example: serious debates over the nature of their public health services are underway in many countries, triggered in large part by the technological advances of the last few decades. These advances have made it possible for humanity to live longer and in better health, but the investments in those gains are considerable. The answer to the four whos are at the crux of the debate. Who benefits — everyone equally at all levels of health? Who pays — regardless of personal health or based on need and ability to pay? Who provides — and determines cost to the user? Who loses — anyone out of work or above some risk level, and who determines who loses?

The problem with HR 3200 and with the arguments put forth to date on its behalf is that they have not systematically and credibly addressed those four questions. In fact, those arguing in support of HR 3200 and health care reform in general have often given contradictory answers to those four questions, undermining their own credibility, given ammo to their opposition, and (justifiably) undermining public support for HR 3200.

Along those lines, the authors also note that in architecting social systems, you face not just the constraints of normal system design — risk, performance, schedule, and cost — but two more: perception vs. facts. They go on to say:

Social systems have generated a painful design heuristic: it’s not the facts, it’s the perception that counts. Some real-world examples: . . .

• One of the reasons that health insurance is so expensive is that health care is perceived by employees as nearly “free” because almost all its costs are paid for either by the the employee’s company or the government. The facts are that the costs are either passed on to the consumer, subtracted from wages and salary, taken as a business deduction against taxes, or all of the above. There is no free lunch.

Again, with great relevance to the current debate over HR 3200 and the whole approach of the House over health care reform, the authors state:

Like it or not, the architect must understand that perceptions can be just as real as facts, just as important in defining the system architecture, and just as critical in determining success. As one heuristic states: the phrase, ‘I hate it’, is direction. There have even been times when, in retrospect, perceptions were “truer” than facts which changed with observer, circumstance, technology, and better direction. . . . In the end, it is a matter of achieving a balance of perceived values. The architect’s task is to search out that area of common agreement that can result in a desirable, feasible system.

Maier and Rechtin end Chapter 5 with some heuristics they consider specific to social systems. Several are those already cited above, but here are a few additional ones (my comments are in brackets):

Success is the eye of the beholder [i.e., the US public] (not the architect [i.e., Congress]).

Don’t assume that the original statement of the problem [e.g., "45 million uninsured"] is necessarily the best, or even the right one. (Most customers would agree.)

In social systems, how you do something may be more important than what you do. (A sometimes bitter lesson for technologists [and Congress] to learn.)

It’s easier to change the technical elements of a social system than the human ones (enough said).

Maier and Rechtin have an entire appendix at the end of the book on heuristics for system-level architecting. Most of these are intended for software and hardware architecting; however, several have bearing for HR 3200 and the general effort for health care reform.

Plan to throw one away; you will anyway.

This comes from Fred Brooks’ classic work, The Mythical Man-Month, and appears to be highly relevant to what’s going on right now in Congress, where both conservative Democrats and Republicans are suggesting that the best approach right now would be to start over again.

In architecting a new [software] program all the serious mistakes are made in the first day.

As someone who has been dealing since 1995 with failed or troubled IT projects, I find that this is the maxim I keep coming back to. I think that the Obama Administration and the Democratic leadership in Congress badly miscalculated public support for rushing sweeping (and unexamined) health care reform into law given the profound economic problems facing the country (not to mention the massive Federal deficits).

Given a successful organization or system with valid criteria for success, there are some things it cannot do — or at least not do well. Don’t force it!

As noted in Part I, HR 3200 is “kitchen sink” legislation, trying to accomplish a variety of changes that are not necessarily related or dependent. I suspect that Obama and Congress would have been far more successful with a series of small, focused bills that had clear goals and clear limits. The problem with HR 3200 is that by trying to cover so much ground, it merely increases the overall size of the opposition — people with objections to a specific portion of HR 3200 find themselves uniting (directly or indirectly) with those objecting to other portions of HR 3200. By recasting HR 3200 into smaller, well-defined chunks, the opposition to any given chunk becomes smaller as well, increasing that bill’s chances of passage.

Group elements that are strongly related to each other, separate elements that are unrelated.

The shorthard version of this in software design is “high cohesion within a module, loose coupling between modules”. This is another argument for breaking up health care reform into smaller, well-defined and clearly-focused chunks.

If you don’t understand the existing system, you can’t be sure you’re re-architecting a better one.

And, I might add, if you don’t understand the proposed system, you can’t be sure it’s a better one. It is unclear that most of the members of Congress who are pushing HR 3200 understand either the current US health care system or HR 3200 itself (and all its implications).

I could include many more maxims here, but you are better off getting Maier and Rechtin’s book and reading it for yourself.

Part III will suggest a different approach to health care legislation using good practices from systems development and software engineering.

[Part II is now up.]

On the occasions where I have reviewed the actual text of major legislation, I have been struck by the parallels between legislation and software, particularly in terms of the pitfalls and issues with architecture, design, implementation, testing, and deployment. Some of the tradeoffs are even the same, such as trading off the risk of “analysis paralysis” (never moving beyond the research and analysis phase) and the risks of unintended consequences from rushing ill-formed software into production. Yet another similarity is that both software and legislation tend to leverage off of, interact with, call upon, extend, and/or replace existing software and legislation.  Finally, the more complex a given system or piece of legislation is, the less likely that it will achieve the original intent.

But there are some critical differences that make legislation design both harder and higher-risk than systems design. (More after the jump.)

Software vs. Legislation

First, software is designed for a target or reference system; you can in theory predict or constrain its behavior, and its behavior is largely repeatable.

Legislation, by contrast, is executed by humans, with wide latitude for interpretation and implementation, as well as misunderstandings, disagreements on meaning, and on-the-fly modifications.

Second, software typically has several layers of independent (non-human) syntactic, semantic, and integration checking that it of necessity goes through before deployment (though plenty of defects can and do  slip through).

Legislation, by contrast, is written in a natural (human) language, with all its gaps, faults, and ambiguities, and with nothing to force error checking in syntax, semantics, and integration; there’s no way of “compiling“, “linking” and doing a test run of the legislation in a limited environment before it becomes the (largely irrevocable) law of the land.

Third, because of the previous two factors, two or more software engineers can typically reach professional agreement on what a given section of source code will do; if they continue to disagree, there are standard tools and methods by which they can objectively demonstrate how the software will behave, either exactly or within general limits.

By contrast, and due to the corresponding factors with legislation, two or more people (legislators, executives, judges, and citizens) can interpret a given section of legislation quite differently, and each may well have a defensible position, due to the potentially wide latitude of and arena for interpretation.

Some Design Flaws of HR 3200

This all comes to mind as I have been reviewing HR 3200, aka the House bill on health care reform. While I am neither a legislator nor a lawyer (though I have worked closely with lawyers for a decade), I am a professional software architect/engineer, and a professional writer, who has worked in the IT field for 35 years. From that point of view, I believe HR 3200 will exhibit profound problems and unintended (or unclaimed) consequences if passed. Here are some of reasons why.

To begin with, HR 3200 suffers from all the problems listed above with legislation. It is written in English, and complex, obscure, jargon-laden English at that. Many of the sections are imprecise and/or incomplete, leaving large amounts of interpretation and implementation to unelected humans. Many of the objections to HR 3200 come from this very problem, including the concern that the ambiguity is deliberate and intended to open doors to politically unpalatable consequences.

HR 3200 is also massive and very complex — over 1000 pages in printed form, with hundreds of sections. For its sheer length alone, it is difficult to understand and interpret, but (as indicated below) there are other factors that make overall comprehension nearly impossible. It also makes after-the-fact revocation or even modification extremely difficult.

Much of HR 3200 makes piecemeal modifications to existing legislation, often with little explanation as to intent and consequences. So, for example:

SEC. 1148. DURABLE MEDICAL EQUIPMENT PROGRAM IMPROVEMENTS.

…

(c) Treatment of Current Accreditation Applications- Section 1834(a)(20)(F) of such Act (42 U.S.C. 1395m(a)(20)(F)) is amended–

(1) in clause (i)–

(A) by striking ‘clause (ii)’ and inserting ‘clauses (ii) and (iii)’;

(B) by striking ‘and’ at the end;

(2) by striking the period at the end of clause (ii)(II) and by inserting ‘; and’; and

(3) by adding at the end the following:

‘(iii) the requirement for accreditation described in clause (i) shall not apply for purposes of supplying diabetic testing supplies, canes, and crutches in the case of a pharmacy that is enrolled under section 1866(j) as a supplier of durable medical equipment, prosthetics, orthotics, and supplies.

Any supplier that has submitted an application for accreditation before August 1, 2009, shall be deemed as meeting applicable standards and accreditation requirement under this subparagraph until such time as the independent accreditation organization takes action on the supplier’s application.’

This happens repeatedly throughout HR 3200; in fact, one entire portion (Division A, Title IV) is labeled “AMENDMENTS TO INTERNAL REVENUE CODE OF 1986″. This makes it difficult — beyond the ambiguities of the language itself — to determine just what is being modified and what the potential implications are.

HR 3200 also suffers in places from what a software engineer would call “spaghetti coding“. In other words, a given section within HR 3200 (and there appear to be hundreds of them; numbers go from 100 up through 2531 and appear in numeric order, but there are many gaps along the way) will reference several other sections elsewhere in HR 3200, both above and below. Furthermore, it often requires careful reading going back pages to see whether a reference to a given section is to a section within HR 3200 itself or a section in existing legislation (such as the Internal Revenue Service code).

HR 3200 also comes across as similar to a “kitchen sink” application, that is, a single piece of legislation that attempts to do far too much.  I will finish Part I with the table of contents for HR 3200 to give you a sense of all that it is attempting to do. Note that these divisions, titles, and subtitles could have been broken up into individual legislation.

Finally, HR 3200 embodies what is commonly known in software engineering as a “big bang” approach to systems development. In other words, HR 3200 attempts a massive and ill-understood (and/or ill-specified) modification to the nation’s health care system (roughly 1/6th of the economy) in one fell swoop. As such, it really represents the worst excesses of the waterfall development lifecycle, with deployment being hard or impossible to reverse.

Part II discusses software architecture and development maxims, laws, and rules of thumb that appear to have application to creating legislation as well.

Part III will suggest a different approach to health care legislation using good practices from systems development and software engineering.

H.R.3200 – America’s Affordable Health Choices Act of 2009 (table of contents)

DIVISION A–AFFORDABLE HEALTH CARE CHOICES

TITLE I–PROTECTIONS AND STANDARDS FOR QUALIFIED HEALTH BENEFITS PLANS

Subtitle A–General Standards

Subtitle B–Standards Guaranteeing Access to Affordable Coverage

Subtitle C–Standards Guaranteeing Access to Essential Benefits

Subtitle D–Additional Consumer Protections

Subtitle E–Governance

Subtitle F–Relation to Other Requirements; Miscellaneous

Subtitle G–Early Investments

TITLE II–HEALTH INSURANCE EXCHANGE AND RELATED PROVISIONS

Subtitle A–Health Insurance Exchange

Subtitle B–Public Health Insurance Option

Subtitle C–Individual Affordability Credits

TITLE III–SHARED RESPONSIBILITY

Subtitle A–Individual Responsibility

Subtitle B–Employer Responsibility

TITLE IV–AMENDMENTS TO INTERNAL REVENUE CODE OF 1986

Subtitle A–Shared Responsibility

Subtitle B–Credit for Small Business Employee Health Coverage Expenses

Subtitle C–Disclosures To Carry Out Health Insurance Exchange Subsidies

Subtitle D–Other Revenue Provisions

DIVISION B–MEDICARE AND MEDICAID IMPROVEMENTS

TITLE I–IMPROVING HEALTH CARE VALUE

Subtitle A–Provisions Related to Medicare Part A

Subtitle B–Provisions Related to Part B

Subtitle C–Provisions Related to Medicare Parts A and B

Subtitle D–Medicare Advantage Reforms

Subtitle E–Improvements to Medicare Part D

Subtitle F–Medicare Rural Access Protections

TITLE II–MEDICARE BENEFICIARY IMPROVEMENTS

Subtitle A–Improving and Simplifying Financial Assistance for Low Income Medicare Beneficiaries

Subtitle B–Reducing Health Disparities

Subtitle C–Miscellaneous Improvements

TITLE III–PROMOTING PRIMARY CARE, MENTAL HEALTH SERVICES, AND COORDINATED CARE

TITLE IV–QUALITY

Subtitle A–Comparative Effectiveness Research

Subtitle B–Nursing Home Transparency

Subtitle C–Quality Measurements

Subtitle D–Physician Payments Sunshine Provision

Subtitle E–Public Reporting on Health Care-Associated Infections

TITLE V–MEDICARE GRADUATE MEDICAL EDUCATION

TITLE VI–PROGRAM INTEGRITY

Subtitle A–Increased Funding To Fight Waste, Fraud, and Abuse

Subtitle B–Enhanced Penalties for Fraud and Abuse

Subtitle C–Enhanced Program and Provider Protections

Subtitle D–Access to Information Needed To Prevent Fraud, Waste, and Abuse

TITLE VII–MEDICAID AND CHIP

Subtitle A–Medicaid and Health Reform

Subtitle B–Prevention

Subtitle C–Access

Subtitle D–Coverage

Subtitle E–Financing

Subtitle F–Waste, Fraud, and Abuse

Subtitle G–Puerto Rico and the Territories

Subtitle H–Miscellaneous

TITLE VIII–REVENUE-RELATED PROVISIONS

TITLE IX–MISCELLANEOUS PROVISIONS

DIVISION C–PUBLIC HEALTH AND WORKFORCE DEVELOPMENT

TITLE I–COMMUNITY HEALTH CENTERS

TITLE II–WORKFORCE

Subtitle A–Primary Care Workforce

Subtitle B–Nursing Workforce

Subtitle C–Public Health Workforce

Subtitle D–Adapting Workforce to Evolving Health System Needs

TITLE III–PREVENTION AND WELLNESS

TITLE IV–QUALITY AND SURVEILLANCE

TITLE V–OTHER PROVISIONS

Subtitle A–Drug Discount for Rural and Other Hospitals

Subtitle B–School-Based Health Clinics

Subtitle C–National Medical Device Registry

Subtitle D–Grants for Comprehensive Programs To Provide Education to Nurses and Create a Pipeline to Nursing

Subtitle E–States Failing To Adhere to Certain Employment Obligations

]]> http://brucefwebster.com/2009/09/07/hr-3200-from-a-systems-design-perspective-part-i/feed/ 7 http://brucefwebster.com/2009/05/28/techno-blindness/ http://brucefwebster.com/2009/05/28/techno-blindness/#comments Fri, 29 May 2009 04:30:37 +0000 bfwebster http://brucefwebster.com/?p=128 A few decades back, when handheld electronic calculators were still pretty neat, someone did a study on the authority people gave to them. As I recall, those conducting the study built some normal-looking calculators that were designed with specific errors in the calculation circuits such that in certain cases the calculators would give wrong answers. The studies in the subject were asked to perform a specific set of arithmetic calculations using these calculators. For some of these calculations, the doctored calculators were certain to give the wrong answer. They were then asked to check the answers by hand and give what they felt was the correct answer.

A large percentage of the subjects — when given a wrong answer by the calculator — would get the right answer when they carried out their calculations by hand. But most of them would assume that they had made a mistake in their manual calculations and that the calculator was correct, and so put down the wrong answer. These individuals couldn’t conceive that that the calculator was giving a wrong answer, and so they would doubt their own by-hand calculations.

Nice story, but old. Why am I telling this? I had a nearly identical experience yesterday, except it didn’t involve a calculator — it involved a GPS unit.

For the last four years, I have lived about 25 miles southeast of downtown Denver. During that period I have driven to and from Wyoming — north of us — multiple times, via the I-25 freeway, so I have a general sense of what lies between Denver and Cheyenne. On the other hand, it’s been probably at least a year since I’ve made that drive, so it’s not exactly fresh in my mind.

Yesterday, I had to drive to Ft. Collins, which is between Denver and Cheyenne (WY), to testify at a trial. I had it pretty fixed in my mind that the trip to Ft. Collins would take about 90 to 105 minutes. I was supposed to be there by 2:30 pm, so I planned on leaving around 12:30 pm, giving myself some slack. Around noon, I got a call from my client’s lawyer, asking if I could be there by 2:00 pm instead. I told him I thought I could make it by then, got things pulled together, and got into the car. I punched the courthouse address into our GPS system — a Magellan Maestro 4250 — asked it to plot my route (“fastest time”), and drove off.

A few miles into my drive, I noticed that the GPS unit showed an arrival time of 2:20 pm. That seemed a bit long to me, and I remembered that I had changed the time zone on the GPS system during my recent three-week trip to California. So I popped the GPS out of its cradle, worked the menus to set the time zone back to Mountain Time, backed out to the map display, and stuck it in its cradle again.

And panicked: the arrival time was now 3:20 pm (which I should have already known, since I was changing from Pacific to Mountain time). I was going to be over an hour late. I might not even be allowed to testify; I was scheduled to be the last witness before my client’s lawyer rested my client’s case. At the best, I was looking at a scolding from the judge and a negative impression on the jury.

Now, with this GPS unit, I’ve occasionally had circumstances where the time-to-destination estimate shifts abruptly, but never by more than a modest number of minutes. I was going a back route, and I thought that the GPS might re-adjust the time estimate when I got onto the E-470 toll road, heading north.

It didn’t. I had picked up a few minutes — the estimated time of arrival was now 3:18 pm — but that bought me very little. While driving at a, uh, vigorous rate of speed on E-470, I canceled the route to the courthouse, then called it up again (on ‘Previous Destinations’) to ensure that I was actually going to LaPorte Road in Fort Collins, Colorado. I was. I selected that destination again, asked the GPS to calculate the route — and again got an arrival time of 3:18 pm or so.

I was dumbfounded. I had been certain that Ft. Collins was an hour and a half, maybe 1:45, from my house, and the GPS system was telling me that it was actually close to three hours. Given how long it had been since my last trip up I-25, I now wondered if I was simply misremembering. I punched the map ‘zoom out’ button repeatedly to ensure that I was going to Ft. Collins. Sure enough — the map showed my course up E-470 to I-25 and then to a point about halfway to the Wyoming border: Ft. Collins.

I was now even more confused. While I was still at home, I had called up Google Maps and plotted a course from my house to the courthouse, and I was sure that it had told me that my driving time was 1:49. But now I was wondering if I had misread the Google Maps page, and it was actually telling me that the distance was 149 miles.

I continued my rapid drive up E-470, hoping against hope that the estimated time of arrival would magically drop an hour or more at some point. It didn’t. Finally, a few miles from the turnoff to I-25 North, I bit the bullet and called the lawyer (via the client’s cell phone) to give him the bad news: that I wouldn’t be there until 3:10 or so (I had picked up a few minutes, though not nearly as many as I had hoped). He was professional on the phone, but I know he felt blindsided — how could I be so stupid (and unprofessional) as to not leave in time to arrive on time? We hung up, and I continued the rapid pace.

I transferred onto I-25 North and saw on the GPS display that my next turnoff was 80 miles away. I wondered for the nth time how I could have been so wrong about the distance to Ft. Collins. Then something bubbled up through my brain: if the Ft. Collins turnoff was only 80 miles away, that meant that I would get off of I-25 around 2:15 pm — it surely wouldn’t take another 45 minutes to get to the courthouse, would it? I tried to think back to the Google map and wondered if Ft. Collins was a really, really long town; if I had to drive another 20 to 30 miles once I was off the freeway. I looked more carefully at the GPS for information on my next exit, the one 80 miles away.

It was in Cheyenne, Wyoming. And I felt hope for about the first time in 45 minutes.

I canceled the route to the courthouse and punched in a new route to Ft. Collins, picking the first street name and number I could punch up. I pressed ‘fastest time’ (the same control I had punched both times for the courthouse) — and got an arrival time of 1:55 pm, well over an hour sooner than the GPS had given for the courthouse. I called up the courthouse address again (from ‘previous destinations’), asked as before for the fastest time — and now got an arrival time of 1:56 pm (instead of 3:10 pm). About that time, I passed a freeway sign indication that Ft. Collins was 37 miles away, so I knew I had the right time this time. I quickly called the lawyer back (actually the client, on his cell phone) and let him know that I would indeed be there by 2:00 pm.

And I was. I testified, left the courthouse, and headed back home at a much more leisurely pace, stopping in Longmont to have dinner with my daughter and grandsons.

This was a genuine bug in the GPS unit that I encountered. All three route calculations to the courthouse used the same address and the same preferences (fastest time), yet the first two times, the GPS was somehow routing me through Cheyenne, Wyoming. Furthermore, when I had zoomed out the map, it did not show me going up to Cheyenne and back; it clearly showed me going only partway to the Wyoming border and then getting off the freeway.

That said, I did exactly what those people in the calculator study did: I trusted the GPS more than my own experience (and more than my recollection of what Google Maps had said). What’s more, I had a set of local and regional atlases in the storage pouch behind the passenger seat; it would have taken me maybe 60 seconds to pull over to the side of the road, pull out an atlas, and verify just how far it was to Ft. Collins. But instead of doing that, I panicked, drove fast, and assumed that the GPS was correct, particularly when I got the same arrival time result the second time.

There was one more point of confusion in all this that likewise could have cleared things up sooner.  When I called the lawyer the first time, to give him the bad news, he asked where I was. I told him that I was on 470, approaching I-25. E-470 is the toll portion of the 470 beltway that goes about 3/4ths of the way around the Denver metropolitan area. I-25, which runs north-south, crosses 470 twice – once north of Denver and again south of Denver. When I told the lawyer how late I was going to be and where I was, he likely assumed that I was at the southern intersection of I-25 and 470, which would put me about 80 miles from Ft. Collins and having to drive right through Denver (heavy traffic, lower speed limits) to get there. Instead, I was about 45 miles and outside of the Denver metropolis, with light traffic and a 75 MPH speed limit.

I’ve had this GPS unit for at least two years. As noted, I’ve had a few glitches in time and route calculation, but never anything of this magnitude. So I let what the machine was telling me override my own experience and knowledge. It is an error in judgment all too frequent when information technology is involved.

Food for thought.  ..bruce..