Back in July 2010, I bought an Acer Aspire easyStore Home Server running Windows Home Server 2003, primarily two solve two problems: (1) run nightly backups of all my Wintel PCs; (2) provide a common media file repository for myself and my wife (who runs Macs). After a few bumps, it worked just fine and has been working fine since.

Except that a few months ago — which at this point may mean something like this past summer or even late spring, given how time flies — I noticed that I could no longer run the WHS Console app from any of my PCs (all running Win7 Pro). Any attempt to do so resulted in an error box with the message:

This computer cannot connect to your home server. Check your network connection and make sure your home server is powered on. If your home server has recently restarted, try again in a few minutes.

Here’s what’s strange: the server itself still shows up in the Network panel, I can still access all the shares, and I can still log onto the server via the Remote Desktop Connection application (and, by the way, run the Console app there, on the server itself). I can ping the server in a console window via its name and its (fixed) IP address. And the automated backups keep happening (though more on this below).

At roughly the same time — and it may have been exactly the same time — I began to get certificate warnings when I used RDC to log into the server. This is a well-known problem with a kludgy work-around (configure RDC to ignore certificate warnings). That problem appears to be associated with Service Pack 1 of Windows 7, and I suspect the Console problem showed up at the same time as well.

However, to date, I have not found a workaround to the Console problem. What’s more, I’m starting to have backup problems. In particular, one desktop system stopped backing up. In an effort to fix that, I uninstalled the WHS Connector Software and attempted to re-install it. No go — I get this error during installation, after entering the server’s admin password:

This operation cannot be completed at this time.

Please try again later. If the problem persists, please contact Product Support.

This appears to be a well-known, if intermittent, problem, but one for which a variety of suggested-but-not-necessarily-effective solutions have been offered. I started to tackle the problem again this morning, trying some of the solutions, and found some other oddities:

• Even though I can see my server (which we’ll call foo-bar for now) in the Network panel, can access it, can ping it (by name or by IP), and can remotely log into it, I cannot bring it up in a browser, e.g., http://foo-bar or even http://<fixed IP address>. That last one is particularly surprising.
• Yesterday, I brought down and then brought back up my entire home-wide network (modem, router, access point, lots of computers and other network-accessing devices). I noticed that the Acer server box took forever (probably 15+ minutes) to reboot itself and reappear on the network.
• I’ve been getting flaky behavior from WHS itself over the past few months when I remotely log in (which I don’t do that often). For example, when I logged in today, I got three standard Windows “unexpected error – send info to Microsoft?” boxes, all associated with the backup utility.
• I did bring up the Computer Management panel on the server, thinking that the inability to browse to the server may depending on something running here. I did find that IIS Admin Services was not running and would not start, even though its two dependencies (RPC and Security Accounts) were, in fact, running.

I suspect that what I may actually need to do is reinstall WHS 2003, but that means that I will have to (for safety’s sake) back up the (non-automated) shares. Not a bad thing to do anyway, but not how I envisioned spending my Thanksgiving weekend.

What struck me while researching on-line is how many people had posts that said, in effect, “I did this and it fixed everything”, followed by posts saying, “I tried that and it still doesn’t work”. Similarly, not everyone running the same systems gets the same problems. It goes to the heart of the underlying complexity, uncertainty, and — to a certain extent — unknowability of the systems upon which we depend.

That said, I welcome suggestions.   ..bruce..

I’ve written here before about the thermocline of truth. The webcomic Partially Clips gives a different, humorous slant; click on the comic to view it in full size. ..bruce..

I’ve joined Google fairly recently after spending nearly a decade at MSFT, and I’m having to unlearn a ton of things I’ve learned at MSFT.

First, I had to unlearn that my opinion doesn’t mean shit. Engineers do, in fact, run Google, and I’m an engineer. A LOT depends on engineers here. Barely anything depends on the management or PMs. The comfortable, asphyxating bureaucracy of Microsoft simply does not exist. It is up to you to define the direction, and execute on it. If you’re good, you will also get other people to execute on it, by means of which you will establish yourself as a leader.

Second, I had to unlearn that my teammates are plotting something behind my back. As far as I can tell a few months in, they aren’t. Or they’re so skilled at it that I don’t see the plot (which after 10 years at MSFT is unlikely). They’re just building a product.

Third, there’s no “jihad” against anyone. Not even Microsoft. People are discouraged from thinking in those terms. No one is trying to “kill the fucking Microsoft”. No one is throwing chairs or calling Ballmer a pussy. People just build their products and services the best they can.

Fourth, there are very few people who can say “no” without motivating their answer with data. The first answer you will hear from anyone (including Legal!) is “yes”. It’s not blind acceptance or anarchy either, it is expected that you will motivate your changes, with data, if necessary. Want to change the way Google runs ads? If your change makes sense and you can demonstrate it, it will be accepted. Search? The same. This one is particularly hard to unlearn – after burying so many great (or at least I thought they were great) ideas because they weren’t _politically_ feasible, sometimes within the same extended team.

And so on and so forth. I wasn’t a bad performer at MS by any means (left the company 5 levels up from where I joined), and as a matter of fact I admire bits and pieces of Microsoft to this day, but Google made me realize just how miserable I was there. I don’t yet feel Google is the ideal place for me either, but one thing is clear – it’s much easier to breathe here, if you know what I mean.

When I wrote The Art of ‘Ware back in 1994, I came away from it with a greater appreciation of why Microsoft had achieved the success that it had. It appears that Microsoft has lost its way. ..bruce..

What is interesting about Holmgren’s hire is that it is modeled after Bill Parcells role at Miami – The Football Czar.  He’s not the head coach and he’s not the GM (who usually handles personnel).  Instead, he is something else.  It is a role that leverages his expertise as a position coach, a head coach and a GM.  One that sees the bigger picture and is able to evaluate players AND coaches from a fresh and more independent perspective.  It is a position created to drive the “rebuilding” of a program – something Miami and Cleveland badly need.  In Parcells’ case, he took an 1-15 team and got it into the playoffs the next year with an 11-5 record.  Part of the Parcells formula is to bring in a core of coaches and players that he trusts and who know his systems, both offensively and defensively.

Be sure to read Curran’s entire post, plus the robust debate in the comments that follow it.

Of course, Ruby and I also used the football analogy a few years ago in suggesting a radically different approach to IT organization and leadership. As we wrote:

The bad news is that, unlike in football, the business and IT sides of a firm don’t always agree on what constitutes a ‘victory’ (even though both sides can usually agree on what a ‘loss’ is, at least in cases of total or significant project failure). Indeed, sometimes they cannot fully agree on what the game is.

Curran’s IT Czar role could in theory solve that; I like his proposals and approach, and, of course, I like the sports team analogy.

The problem is, it’s hard to see how upper management would treat or perceive it as being any different from the current CIO role. Both as a consultant and as an expert witness, I’ve had lots of opportunity to see just how  constrained the CIO slot can be in large organizations. A lot of the blame for that rests squarely upon the CEO and other CxO leaders, both in terms of whom they select for the CIO job and how they define/constrain that job. CIOs tend to be selected from business-types with some technical background, as opposed to technical types with some business background. Curran’s analogy (e.g., Bill Parcells) suggests that the czar be someone who has been down in the trenches (e.g., position and head coach) and knows what day-to-day IT development requires, but that’s not who usually gets picked.

The closest model to an IT Czar that I’ve seen that has worked were the various corporate Y2K ‘czars’ who were appointed 10+ years ago to save the officers and directors from any Y2K liability, and thus were given lots of power and pretty free rein. (A joke from those times — Q: What’s the difference between a terrorist and a corporate Y2K director? A: You can negotiate with the terrorist.) An IT czar appointed to ‘turn around’ an organization’s IT efforts would have to be given that same power and freedom. That is not likely to happen unless, as with Y2K, the corporate officers and directors see themselves at risk — professionally, legally, financially and/or personally. Otherwise, as noted above, the “IT Czar” would be just another CIO.  ..bruce..

This naturally intrigued me, since for the last 15 years, I have been writing, consulting, lecturing, and testifying about troubled and failed IT projects. While there are indeed tremendous financial losses due to late and failed IT projects, the figures Sessions gives seem much too large to me, and so I decided to do this critique of his analysis.

Sessions is good enough to provide the basis of his estimates and calculations, including footnotes. But that’s where some of the problems start. For example,  on page 3, Sessions cites (his footnote ’02′) to the US Budget, Fiscal Year 2009, Analytical Perspective (PDF), p. 169, for information on “at-risk” or failed IT projects, specifically:

• “According to the 2009 U.S. Budget [02], the failure rate is increasing at the rate of around 15% per year. If this trend continues, within another five years or so a total IT meltdown may be unavoidable.” (p. 3)
• “According to the 2009 U.S. Budget [02], 66% of all Federal IT dollars are invested in projects that are ‘at risk’. I assume this number is representative of the rest of the world.” (p. 3, in “Calculating the Cost of IT Failure” box)
• A large number of these ['at risk' projects] will eventually fail. I assume the failure of an ‘at risk’ project is between 50% and 80%. For this analysis, I’ll use the average: 65%.”

These three statements run into immediate problems. First, and relatively minor, Sessions gets his page number wrong: he’s citing “page 169″ of the Analytical Perspective document, but there is no discussion whatsoever on page 169 of that document about IT projects. However, page 157 of that document (which happens to be page 169 of the PDF document) does start a section titled “INTEGRATING SERVICES WITH INFORMATION TECHNOLOGY”, so I presume that Sessions made the simple mistake of using the PDF page count rather than the document’s actual page numbering.

Even so, serious problems remain with Sessions’ citations and analysis.

Page 157 of the Analytical Perspective document does not say what Sessions claimed in the two comments above. I have not been able to figure out where Sessions gets his figure for “the failure rate increasing around 15% per year” from the cited US Budget Analytical Perspective document, much less his conclusion that “if this trend continues, within another five years or so a total IT meltdown may be unavoidable.” As far as I can tell, the Analytical Perspective document does not talk about failed IT projects at all, much less the increase in failure rates.

Furthermore, the phrase “the failure rate increasing around 15% per year” is itself ambiguous and may not be that significant. To start with an arbitrary number, assume that 100 projects “fail” in a given year. If “the failure rate [is] increasing around 15% per year”, then that means that 115 projects would fail the next year, and 132 projects would fail the year after that. But unless we know both the actual number of failed IT projects and the total number of IT projects in that same year, Sessions’ figure tells us nothing. If there’s only 150 IT projects total, then the 15% failure rate increase becomes very significant; if there’s 1000 IT projects total, then we’re many years away from Sessions’ threatened “meltdown”.

Sessions also ignores or confuses the failure rate for new projects vs. the systems already deployed. In other words, the failure rate for new systems development says very little about the continued functionality of existing, deployed systems now in use. While there are occasions (most notably Y2k, now a decade behind us) where existing IT systems just won’t function or function properly if they aren’t fixed or replaced, by and large both governments and private concerns have gotten along remarkably well for years or even decades with antiquated systems

As for Sessions’ second statement, there is a table on page 158 that may represent the basis for it:

As can be seen in the FY 2009 column, 66% (535 out of 810) of the FY 2009 “Major IT Investments” are projects that are “Not Well Planned and Managed”. Note that this table does not (as Sessions infers) indicate Federal dollars but rather actual projects; that is, in FY 2009, there are 810 projects listed as “Major IT investments”, of which 535 are designated as “Not Well Planned and Managed”. The previous page appears to indicate that these projects represent $27 billion, which is roughly 38% of the proposed Federal IT budget — not a great figure, but still almost half of the 66% that Sessions claims.

What’s more, supplementary data (PDF) for the FY 2009 Analytical Perspective makes it clear that the US Government’s designation of such projects — which puts them on a “Management Watch List” (WML) — has reduced the risk of such projects during each fiscal year:

Note that in FY 2007 and 2008, the number of IT projects designated as “Not Well Planned and Managed” shrunk significantly during the year (from Q1 to Q4) without a proportional shrinkage of the overall number of major IT projects. In other word, it appears that the government’s efforts to remove such projects from the “Not Well Planned and Managed” category is relatively successful. And the actual US IT budget dollars at risk at the end of each of those fiscal years ($4.2 billion for FY 07, $8.6 billion for FY 08)  is a much smaller percentage (6.5% and 13%, respectively) of the Federal IT budget for each of those years ($64.2 billion for FY 07 (XLS), $66.4 billion for FY 08 (XLS)).

Sessions then states that “I assume this number [66% of all Federal IT dollars being at risk] is representative of the rest of the world.” There are numerous problems with this assumption, starting with the fact that the 66% figure is wrong; in fact, the actual “at risk” (his term, not the US Government’s) percentage of the IT budget at the end of FY 07 and FY 08 were, as noted above, 6.5% and 13%, respectively.

Sessions’ error here is significant, since he goes on in several places (cf. page 4) to cite his use of the % of the total IT budget as being significant, when he’s not talking about the total IT budget at all.

Furthermore, it is unclear whether his phrase “the rest of the world” means all other national governments, or all other entities doing IT project development. It seems to be the latter, though it’s hard to tell from his statements. On the other hand, I have spent years consulting with corporations on troubled projects, and I can tell you that they do not have 66% of their IT budgets devoted to “at risk” projects. In fact, the majority of corporate IT budgets are devoted to maintenance of existing systems, not new and risky projects (cf. here, here, here, and here, as simple examples).

As noted, Sessions then assumes that the failure rate for “at risk” IT projects is 65%, which means that (as he says) “I am calculating that 43% (.65 x .66) of the total IT budget” is devoted to failed projects. At this point, his figures become nonsensical, as they are derived both from misreadings and lack of complete information about the Federal IT budget and projects. To wit:

• The 535 “not well planned and managed” IT projects in the US FY 09 budget only represent 38% of the total IT budget, not 66% as Sessions mistakenly states.
• In the two previous years (FY 07 and FY 08), the number of IT projects labeled as “not well planned and managed” dropped during the course of each year (see the 2nd table above). In FY 07, it dropped from 263 projects in Q1 to just 84 in Q4, which means that 69% were moved off of the “not well planned and managed” list during the year. Likewise, in FY 08, it dropped from 346 projects in Q1 to 134 projects in Q4, a drop of 61%. This directly contradicts Sessions’ assumption of a 65% failure rate for projects in the “not well planned and managed” category.
• The FY ’09 Analytical Perspective says nothing about actual failed projects, as far as I can tell.

Sessions then goes on to make further out-of-his-hat assumptions regarding “direct and indirect costs”. He cites an example of the IRS (an agency long troubled by IT woes) and notes a lost opportunity based on fraudulent tax returns due to the system not being operational. He projects a loss over two years ($1.788 billion), compares it to the cost of the failed modernization ($185 million over a ten-year period), and calculates an indirect costs ratio of 9.6 to 1. He then decides — with no other documentation or analysis whatsoever — that the universal ratio of indirect to direct costs for a failed IT project ranges from 5:1 to 10:1, and uses the “average” of 7.5:1.

There are so many problems here that I scarce know where to start. For starters, the term “average” assumes an even distribution of ratios from 5:1 to 10:1 and does not recognize any ratios lower than 5:1. I’ve seen many failed projects that had much lower ratios of “indirect” to “direct” costs, since the firm simply continued to operate using the existing systems, and the “lost opportunity” for not having the new system in place was relatively small.

More importantly, the IRS gets to collect taxes from the entire US: $2.5 trillion in tax collections each year. Using the IRS as a baseline makes little sense for most other government agencies, and even less sense for most corporations and non-government organizations (NGOs), because most IT systems in most organizations (government or private) do not have the ability to generate such magnitudes of revenue, period.

Indeed, there is a long-standing controversy within IT management circles as to whether a new computer system can be relied upon to provide any significant return on investment (ROI), or whether it exists merely to “keep up with the competition”.

Sessions concludes his section on calculations thusly (p. 5, emphasis his):

Of course, these calculations are estimates. I recommend you don’t get overly focused on the exact amounts. I could be off by ten or twenty percent in either directions. The real point is not the exact numbers, but the magnitude of the numbers and the fact that the numbers are getting worse.

Unfortunately, Sessions is fundamentally wrong in his numerical analysis, and his numbers are off by far more than “ten or twenty percent”. For the Federal Government alone, they are off by almost  a full order of magnitude (10x), due to his critical errors both on the percentage of the Federal IT ’09 budget “at risk” (it’s 38%, not 66%) and the number of “at risk” projects that fail (he says 65%; the US government numbers for FY 07 and 08 show that only 35% of the projects — representing just 6.5% to 13% percent of the Federal IT budget — were still “at risk” at the end of each fiscal year, and it gives no figures that I can find for actual failed IT projects).

Furthermore, his projection of the (erroneous) 66%-of-IT-budget-at-risk figure on the rest of the world is just wrong, especially in corporations and business (which spend vastly more on IT than the US government). In those organizations, maintenance costs dominates, and the percentage of the IT budget devoted to new projects tends to be small (20% or less), with an even smaller fraction of that representing “at risk” projects.

I may comment more on Sessions’ paper, but my conclusion here is that his estimate of $500 billion/month in lost direct and indirect costs due to IT systems failure just does not hold up, in my opinion.  ..bruce..

Each technology is on its own product lifecycle, which may or may not match with your organization’s business and development lifecycles. In particular, there are certain cycle mismatch patterns that commonly occur in organizations looking to adopt new technologies. I’ve labeled four such mismatch patterns: firefly, underdone, conveyer belt, and landfill. Each is worth examining.

Go read the whole thing.  ..bruce..

Now, and again courtesy of Rand Simberg, comes this brilliant video, made recently by frustrated workers within NASA to show how efforts to innovate and improve a troubled project (can you say “Ares”, boys and girls?)  get blocked by middle managers:

Simberg actually pointed to a post by Wayne Hale, a long-time NASA manager, on an official NASA website blog. Hale felt that many of these problems had already been addressed and was surprised (or Simberg put it, “shocked, shocked”) to find them still pervasive at NASA:

Recently I had a couple of events which affected my thinking on this.  I have been out of the Shuttle Program manager job for almost a year now and a trusted coworker just a week ago told me that people in his organization had been prevented from giving me important alternative choices for some program choices that occurred a couple of years ago.  This was staggering. It was happening right in front of me and I was totally unaware that people – who I trusted, who I hoped would trust me – kept their lips sealed because somebody in their middle management made it clear to them that speaking up would not be good.

Astounding.

About two weeks ago an activity that Mike Coats started at JSC had an all day report out period.  The Inclusion and Innovation Council was to propose ways to improve innovation at NASA.  Various teams reported out, including one team of young employees who has the task to talk about the barriers to innovation at NASA — specifically at JSC.

The video attached was their result.  I found it extraordinarily funny and not at all funny.  These young people have obviously found themselves in situations RECENTLY in which managers at various levels applied sociological and psychological pressures to keep them from bringing ideas forward.

I am convinced that if we asked the managers who were the models for this little morality play whether they stifled dissent or welcomed alternate opinions, they would respond that they were welcoming and encouraging.  Probably because they have that self image.

But actual behavior, not inaccurate self perception, is what we really need.

Watch the video, read Hale’s post, and then ask yourself: how many of these problems affect innovation and product development in both organizational and commercial IT development groups? ..bruce..

• Performing IT Project Triage
• Pulling the Plug on IT Project

Next up: how to deal with personnel issues.  ..bruce..

And if you have read them, you should probably re-read them.  ..bruce..

My two latest Baseline columns talk about ways to retain IT staff. The first column talks about making an effort to align your staff’s individual professional goals with your organization’s goals. The second column talks about how to improve your IT staff while encouraging them to stay with your firm.

As always, feedback is welcome here or there.  ..bruce..