Bruce F. Webster http://brucefwebster.com Making information technology work since 1974. Thu, 26 Jun 2008 00:18:28 +0000 http://wordpress.org/?v=2.5.1 en The decline in computer science students (part 2) http://brucefwebster.com/2008/06/24/the-decline-in-computer-science-students-part-2/ http://brucefwebster.com/2008/06/24/the-decline-in-computer-science-students-part-2/#comments Tue, 24 Jun 2008 22:27:35 +0000 bfwebster http://brucefwebster.com/?p=49 I previously discussed the up-and-down cycle of college enrollment in computer science and related fields. More accurately put, there have been two large peaks in computer science enrollment: one in the mid- to late 1980s (which happens to be when I was teaching CS at Brigham Young University) and another right around the turn of the 21st century.  Here’s the CRA chart I included in that previous post (click on the chart to see a larger version):

Back in 1985-87, while I was teaching at BYU, I mentioned to my friend Wayne Holder — one the finest software engineers I’ve ever known — that students at BYU could no longer simply declare their major to be Computer Science; instead, they had to take certain prerequisites, apply to the CS department, and be accepted. Wayne thought that was too complicated. He suggested that the prospective candidate be put into a room with (a) a bowlful of money and (b) some really nifty hardware and software. The candidate could then choose either to grab a handful of money and leave or to hang out and play with the computer gear; those who chose the latter would be admitted to the program.

I think Wayne was dead on, and this article in Computerworld (hat tip to Slashdot) tends to support that, though the survey quoted in from the United Kingdom rather than the United States:

Responses from nearly 2,000 undergraduates across the UK showed that most students think the IT sector has a bright future with good prospects for highly paid jobs.

But over 60% of non-computing students do not wish to enter the sector because they think it will be boring.

I’ve written before that talent is a key factor in IT personnel issues, and only a small portion of the general population appears to be talented in IT. People who have little or no aptitude for IT are likely to find it boring at best and confusing at worst.

However, that natural aversion to IT has been overcome at least twice in the last 30 years. The first time was in the mid-1980s and was largely a response to the explosive growth of the personal computer industry, led by Apple, IBM and Microsoft, but including many, many firms making both hardware and software. I wrote for BYTE Magazine back then, and individual issues of BYTE ran anywhere from 300 to nearly 600 pages, due to the sheer volume of ads. My observation as a CS instructor at BYU was that many of our students had come into the program thinking they were going become rich and/or famous, like Steve Jobs or Bill Gates.  They viewed computer science the same way my fellow undergrads a decade earlier had looked at law or med school. Hence the tremendous run-up in CS enrollment, not just at BYU but all across the United States.

Then came the First Tech Crash, which hit around 1988 — helped along, if not outright triggered by the stock market crash in October 1987 — and lasted into 1991 or so. Large numbers of hardware and software companies went out of business, and the personal computer market pretty much narrowed down to IBM and a small number of IBM PC clone manufacturers, with Apple treading water (at best). The chart above shows how CS enrollment mirrored that crash. By the early 1990s, the joke in the IT industry was: “Do you know what the status symbol of the 90s is? A job.”

CS enrollment nationwide was pretty flat from 1991 to 1997, and down at a level that you’d have to go back to 1981 to match. Most likely, people going into computer science at that time were — like me, all the way back in 1974 — going into it because we liked the field, not because we thought we’d be rich.

By 1998, however, the “dot-com boom” had become visible enough to start driving CS enrollment up again. There was an enormous demand for software engineers, with a lot of venture capital to back it up — news articles reported programmers being recruited out of high school, and CS graduates were getting large salaries and signing bonuses. Beyond that was the vision of the “nerd lottery” (to use Bruce Henderson’s phrase): dot-com startups would go public, and many of the startup’s employees (right down to receptionists) would walk away multi-millionaires. Mainstream corporations tried to get in on the dot com boom as well, starting various e-commerce and internet intiatives.

In just about this same time period, the Year 2000 (Y2K) problem got everyone’s attention, and even those organizations, both commercial and governmental, that kept the dot-com craziness at arm’s length found themselves having to do exhaustive testing and remediation of their IT systems from top to bottom. Business and government in the United States would end up spending $110 billion on Y2K remediation, all in just a few years.

As the chart shows, CS enrollement skyrocketed again, nearly tripling from 1997 to 2003, largely due to the combination of these two factors.  Unfortunately for those students, Y2K remediation largely finished up almost at the same time the Second Tech Crash (or “Dot Com Crash”) started, namely March 2000. The NASDAQ stock index peaked at its all-time high value of 5048.62 on March 10, 2000, a 100% increase over what it had been just a year earlier. (Stop and think about that: what if the Dow Jones Industrial Average were to hit 24,000 a year from now?) It was a classic bubble, and now it was popping, or at least deflating; the NASDAQ index currently trades at less than half that value. (Note that the DJIA is up roughly 20% — and was up over 30% earlier this year — from its value on that same date eight years ago.)

This tech crash was far more brutal than the first one. The IT employment marketplace was flooded with massive numbers of IT engineers who were no longer needed, one way or the other, and even talented IT engineers had a hard time getting visibility over the sheer number of warm bodies out there.  But it took a while for that feedback to get back into the colleges and universities; enrollment continued to climb until about 2003 but appears to have been slumping since then (see the chart above) and could actually drop back nearly to where it was when I graduated with my own CS degree some 30 years ago.

In other words, the real issue isn’t why CS enrollment is declining; the question is why did it ever climb so high in the first place? And it’s pretty clear that it tracks the two major bubbles of the past 30 years: the personal computer boom in the mid-1980s and the dot-com/Y2K boom of the late 1990s. After each bubble deflates, CS enrollment sinks back to its “natural” level, based on the distribution of IT-related talent and inclination in the general population.

The problem, however — as I first noted over 12 years ago — is that this “natural” level isn’t enough to supply sufficient IT talent for successful IT develompment and deployment in all the businesses, vendors, government agencies and other organizations that need it.

In my opinion, there is no shortage of IT engineers — particularly not after the vast numbers drawn into the industry due to Y2K and the dot-com boom — there’s just a shortage of talented ones. This is why you get conflicting claims and statistics about “personnel shortages” in the IT industry (cf. here vs. here, as well as the battle over raising the limit on H-1B visas and the offshoring debate).

The various attempts to “boost” CS enrollment at colleges and universities will have only a small effect on that talent shortage; for the most part, it will likely bring additional people into the IT industry who lack the talent or inclination to do well there.  In other words, it won’t solve our IT problems at all.  ..bruce..

]]>
http://brucefwebster.com/2008/06/24/the-decline-in-computer-science-students-part-2/feed/
Issue: metrics for tester productivity? http://brucefwebster.com/2008/06/20/issue-metrics-for-tester-productivity/ http://brucefwebster.com/2008/06/20/issue-metrics-for-tester-productivity/#comments Fri, 20 Jun 2008 22:33:54 +0000 bfwebster http://brucefwebster.com/?p=48 In response to my Baseline columns on metrics (Part 1, Part 2, and Part 3), I received the following e-mail:

I read your column with great interest as I’m involved on an IT project to measure productivity. May I ask you a quick question? Are there any mature metrics that can measure tester productivity improvement month by month and accurate to 1%?

Here’s the response I sent back:

Well, for starters you have to define what you mean by “tester productivity.” Number of test scripts run? Number of defects found?  Number of defects closed? Number of defects reopened? (And do you weight the “defects found/closed/re-opened” by criticality and/or severity?) Number of reported defects replicated? Number of hard-to-replicate, yet critical/severe defects that can now be replicated (and thus fixed)? Some combination (possibly a weighted function) of all of the above?

In other words, what is it exactly that you’re trying to accomplish? To make your testing team more effective? More efficient? To shorten the test cycle? To spend less on testing? To close more defects (and defer fewer open ones) for each system release? To have fewer defects discovered after a system release? Jerry Weinberg says that “quality is value to some person.” Who are the people you’re worrying about, what qualities — functionality, performance, reliability, etc. — do they value, and to what extent?

Once you’ve defined all that, there still remains the question as to whether you can measure that to a 1% accuracy (or even a 10% accuracy) month over month, and still preserve any meaning in that measurement. It’s possible (and common) in metrics to have “false accuracy”  — you believe you’re actually measuring something to a certain precision, but you’re mostly just reading random or insignificant noise at that level.

Finally, we come back (as always) to Weinberg’s law of metrics: that which can be measured can be fudged (or exploited). For example, read this story over at the Daily WTF:  The Defect Black Market.

Hope this is of some help, though I tend to doubt it. :-)

Thoughts from the rest of you?  ..bruce..

]]>
http://brucefwebster.com/2008/06/20/issue-metrics-for-tester-productivity/feed/
Latest column: IT project metrics, part 3 http://brucefwebster.com/2008/06/19/latest-column-it-project-metrics-part-3/ http://brucefwebster.com/2008/06/19/latest-column-it-project-metrics-part-3/#comments Thu, 19 Jun 2008 23:04:10 +0000 bfwebster http://brucefwebster.com/?p=47 My newest Baseline column is up: “Lies, Damned Lies, and Project Metrics (part 3)“. In it, I wrap up my discussion on IT project metrics, outlining a possible approach using instrumentation and heuristics.  Go check it out.  ..bruce..

]]>
http://brucefwebster.com/2008/06/19/latest-column-it-project-metrics-part-3/feed/
Anatomy of a runaway IT project http://brucefwebster.com/2008/06/16/anatomy-of-a-runaway-it-project/ http://brucefwebster.com/2008/06/16/anatomy-of-a-runaway-it-project/#comments Mon, 16 Jun 2008 16:43:53 +0000 bfwebster http://brucefwebster.com/?p=46 The following document is the actual text — carefully redacted — of a memo I wrote some time back [i.e., several years ago] after performing an IT project review; names and identifying concepts have been changed to preserve confidentiality (and protect the guilty). The project in question was a major IT re-engineering effort for a mission-critical system; at the time I did this review, the project had been going on for several years and had cost millions of dollars; it would eventually be canceled and the work products abandoned. The memo itself provides an interesting glimpse into just how a major IT project can go so far off the tracks that nothing useful is ever delivered.

Note that the “ABC” consultants were a small part of the overall project team and had been brought in relatively late by “BigFirm” in an attempt to get the “FUBAR” project into production; they neither initiated nor managed the project. [NOTE for those of you who have written or done Google searches: "Bob Winsom", like all the other names in the memo as transcribed below, is a pseudonym.]

========================================

CONFIDENTIAL MEMORANDUM — EYES ONLY

Over the past two weeks, I’ve conducted confidential off-site group interviews with all of the ABC consultants working on the FUBAR project. I did this at [ABC manager's] request, after a few of these consultants spoke privately about FUBAR with him. The feedback was consistent and raises serious doubts about whether the FUBAR project, as currently pursued, can ever yield a successful production deployment.

This report groups those comments into several broad areas. It is relatively unfiltered and extremely direct (no withholding). It represents the private comments of ABC consultants who have little to gain and possibly much to lose by being so blunt. These are not the whinings of purists picking nits. These are the grounded assessments of top-notch IT professionals who have among them a century or two of experience bringing projects to completion — particularly those involving [specific IT] technology — and who are down in the FUBAR trenches every day.

QUALITY OF WORK AND EFFORT

ISSUE: Several consultants said — and the rest pretty much agreed — that far too many of the deliverables, artifacts, and activities (e.g., algorithms, source code, system configuration, design/architecture documents, testing, defect tracking, scheduling etc.) are substantially below any acceptable professional standards and represent a profound threat to FUBAR ever going into production.

EXAMPLES: The code base is very fragile. A lot of it is bad old code that BigFirm didn’t have time to rewrite two years ago, but now is five times its original size and even worse. One consultant said he took a code listing, picked pages at random, and found problems on every page he selected. There is pervasive hard coding of what should be adjustable parameters or at least meaningfully named constants (e.g., # of [key items] hard-coded throughout with the literal value ‘3′, a constant named ‘ninety_eight’). Builds take all night. App releases don’t run acceptably, if at all, in a production environment. Developers check in files that won’t even compile.

RISKS: The FUBAR project keeps being touted as a world-class development team, but it is not producing world-class, or even minimally-professional, results. This already shows up in the project delays and quality issues of the releases to date. What the team is producing will not only be very difficult to support and modify, it will in all likelihood be unusable, resulting in a complete failure of the FUBAR project.

PROJECT PLANNING AND EXECUTION

ISSUE: Project planning and execution is all to often poor or missing completely. Milestone dates, usually unrealistic if not impossible, are based on political considerations or wishful thinking, not bottom-up grounding. Necessary and useful activities are delayed or canceled with the justification “We have no time for that”, but the project phase ends up taking as long or longer than if the activities had been carried out. Dates are set, but nobody scrambles until the last minute. Risks are not actively tracked or managed.

EXAMPLES: Count how many times FUBAR ever produced a production-quality deliverable on anything close to a scheduled date. Even the current effort probably requires a year to get something into production, but the schedule says four months. Managers create work tasks, but then never track progress or completion. One ABC consultant created a risks document; Bob Winsom [BigFirm's FUBAR project manager] took it over, and no one has seen it since.

RISKS: FUBAR is massively late. You lose credibility and influence.

QUALITY ASSURANCE AND PROCESS

ISSUE: Quality assurance appears to be low-priority concept within the FUBAR project. In the opinion of several consultants, the current person in charge of it is not particularly strong or competent. There appears to be a systemic inability to establish good testing criteria and methods to gauge FUBAR’s progress toward production. What software lifecycle process is in place is often not followed. No independent group or person acts as the ‘gatekeeper’ to judge acceptability and control release into production.

EXAMPLES: [Key process] calculation — the core of BigFirm’s business and profits — was being (and may still be) done incorrectly in FUBAR; it had never been previously checked for correctness through all these years. Likewise, performance expectations have been based on the presumption of FUBAR distributed over multiple systems, processors, and threads, yet no one ever tested to see if those implementations would work until recently — and they didn’t. The build environment needs to be overhauled. The defect tracking process is poor, particularly the practice of writing up defects not against the current release but the release in which the defect is scheduled to be fixed — so as to keep the number of defects down for the current release.

RISKS: BigFirm leaves itself open to potential liabilities, not to mention crippling its own core business. In the meantime, the effort to transition directly into the Rational Unified Process (RUP) is not being given sufficient time and will likely grind development to a halt.

ARCHITECTURE

ISSUE: FUBAR doesn’t have a viable, consistent architecture. The irony is that FUBAR itself is not a big, complex problem; it is a relatively straightforward problem that has been made big, complex, and possibly unsolvable in the current implementation. Initiatives to rearchitect are started, abandoned due to “schedule pressure”, then restarted months later.

EXAMPLES: The project has gone through a series of architects, who have either left or been asked to leave. While they are here, they usually are neither listened to nor given the authority to be an architect. Technical decisions are made by people lacking the background, such as Bob Winsom, who may fancy himself an architect and was quoted as saying, “I haven’t found an architect I like yet.”

RISKS: FUBAR never stabilizes enough to go into production for any length of time, or if it does, proves to be extremely difficult to support or enhance.

APPLICATION PERFORMANCE

ISSUE: FUBAR was never properly architected and designed for the performance required. There is a current effort to increase performance after the fact, but the implementation makes that impossible. To make things worse, developers are having to scale the performance of and debug a seriously flawed application at the same time, making it very hard to stabilize the application.

EXAMPLES: Two consultants rewrote the 140,000 lines of [original obscure language] into 4200 lines of Java. The Java version runs as fast on a laptop PC as the original version runs on a high-powered UNIX server.

RISKS: Despite heroic efforts (that will probably make the application even more difficult to modify and support) and lots of hardware, FUBAR will probably reach some fraction of the currently-desired performance — possible as little as 15% to 20% of that required, possibly as much as 80% — and then go no further.

STAFFING

ISSUE: Many of the people involved in FUBAR — developers, testers, team leads, managers — lack the qualifications, insight, or experience to make FUBAR a success. The project is overstaffed for the actual complexity of FUBAR, possibly for political reasons (i.e., promotion/stature based on the number of people supervised).

EXAMPLES: Many of the examples listed above reflect this problem.

RISKS: This problem leads in part to all the issues previously listed: poor quality of work, poor quality assurance, poor scheduling and delivery, poor architecture, poor application performance. Besides the potential failure of FUBAR itself, this issue tends to be self-intensifying — that is, the qualified people become frustrated and leave (or are hard to recruit in as replacements), while those less qualified or capable stay around. [A reference to the "Dead Sea effect" written many years ago.]

[BrandName team management approach] PRINCIPLES

ISSUE: Mid-level management tells the developers that mood, sincerity, and commitment are everything, and that with them “we can accomplish anything.” At the same time, the principle of granting sincerity appears to be used to justify a lack of accountability and consequences.

EXAMPLES: Repeated statements in team meetings, one-on-ones, and so on.

RISKS: Loss of credibility. Such assertions don’t hold water. I can be in a great mood and have a team of very sincere and committed people, but if we try to build a commercial airliner without the proper expertise, requirements, engineering, materials, and testing, the plane will crash and people will die, assuming it ever gets built and off the ground (which is extremely unlikely). The fallacy that software is somehow different is just that — a fallacy, and one that costs corporations millions (if not billions) of dollars a year in missed schedules and failed projects. When it comes to engineering, sincerity and commitment, while important, can never substitute for expertise and quality of work.

INTELLECTUAL HONESTY

ISSUE: There isn’t enough intellectual honesty within the FUBAR project. Managers reject or explain away bad news and real problems, looking instead for people who will tell them what they want to hear.

EXAMPLES: Several developers and team leads have sought to escalate these issue and concerns up the management chain, but the issues appear to always get stopped, usually at Bob Winsom. [The "thermocline of truth", with a very discrete boundary.] The FUBAR project is represented as something that has never been done before, and the staff as a world-class development team.

RISKS: The lack of intellectual honesty in project management is a form of codependency and enabling that is all too easy to fall into. Unfortunately, reality eventually intrudes, and when it does, you run the very real risk of looking dishonest or incompetent.

CLOSING REMARKS

As I said, this is a very blunt (and very confidential) memo. It represents the opinions, experiences, and observations of these ABC consultants, and there are undoubtedly points with which you take issue or disagree. Do not let that blind you to the fundamental reality that there are some profound problems and flaws with the FUBAR project that will not go away until the project team acknowledges and addresses them. Indeed, it will be hard enough to make them go away even then.
========================================

Kind of scary, isn’t it? The interesting part was that BigFirm had implemented, corporate-wide, a “team management” methodology (from an outside firm) that was based on “mood, sincerity, and commitment”. As an overall corporate management approach, it might well have been effective; I just don’t know. But BigFirm thought that it would also solve their IT problems.

Nope, it didn’t. ..bruce..

[Speaking of project failure -- I have my first three "Surviving Complexity" columns up at Baseline, talking about IT project metrics, why they're so tough to define, and one possible approach.]

[UPDATED 06/25/08: If you think the project above is bad, take a look at this one.]

]]>
http://brucefwebster.com/2008/06/16/anatomy-of-a-runaway-it-project/feed/
Latest column: more on metrics http://brucefwebster.com/2008/06/13/latest-column-more-on-metrics/ http://brucefwebster.com/2008/06/13/latest-column-more-on-metrics/#comments Fri, 13 Jun 2008 21:48:05 +0000 bfwebster http://brucefwebster.com/?p=45 My newest Baseline column is up: “Lies, Damned Lies, and Project Metrics (part 2)“. In it, I talk about why it’s so hard to apply metrics to IT project management and begin to suggest an approach.  Go check it out.  ..bruce..

]]>
http://brucefwebster.com/2008/06/13/latest-column-more-on-metrics/feed/
Gender differences in coding styles? http://brucefwebster.com/2008/06/09/gender-differences-in-coding-styles/ http://brucefwebster.com/2008/06/09/gender-differences-in-coding-styles/#comments Mon, 09 Jun 2008 17:10:53 +0000 bfwebster http://brucefwebster.com/?p=44 In my earlier post on the “thermocline of truth“, I wrote:

Second, IT engineers by nature tend to be optimists, as reflected in the common acronym SMOP: “simple matter of programming.” Even when an IT engineer doesn’t have a given subsystem completed, he tends to carry with him the notion that he whip everything into shape with a few extra late nights and weekends of effort, even though he may actually face weeks (or more) of work. (NOTE: my use of male pronouns is deliberate; it is almost always male IT engineers who have this unreasonable optimism. Female IT engineers in my experience are generally far more conservative and realistic, almost to a fault, which is why I prefer them. I just wish they weren’t so hard to find.)

Now, a post over at the Wall Street Journal cites what I think is a more controversial (and harder to support) claim — by a female VP of Engineering — that female programmers tend to write clearer and better-documented code than male programmers:

Emma McGrattan, the senior vice-president of engineering for computer-database company Ingres–and one of Silicon Valley’s highest-ranking female programmers–insists that men and women write code differently. Women are more touchy-feely and considerate of those who will use the code later, she says. They’ll intersperse their code–those strings of instructions that result in nifty applications and programs–with helpful comments and directions, explaining why they wrote the lines the way they did and exactly how they did it.

The code becomes a type of “roadmap” for others who might want to alter it or add to it later, says McGrattan, a native of Ireland who has been with Ingres since 1992.

Men, on the other hand, have no such pretenses. Often, “they try to show how clever they are by writing very cryptic code,” she tells the Business Technology Blog. “They try to obfuscate things in the code,” and don’t leave clear directions for people using it later. McGrattan boasts that 70% to 80% of the time, she can look at a chunk of computer code and tell if it was written by a man or a woman.

I’m not sure that I could make the same claim without some serious research into a broad range of coding samples. But the article is worth reading for the comments that follow it, which (as you can imagine) are quite intense.  ..bruce..

]]>
http://brucefwebster.com/2008/06/09/gender-differences-in-coding-styles/feed/
New column for Ziff Davis: “Surviving Complexity” http://brucefwebster.com/2008/06/05/43/ http://brucefwebster.com/2008/06/05/43/#comments Fri, 06 Jun 2008 00:03:06 +0000 bfwebster http://brucefwebster.com/?p=43 As I’ve mentioned here before, I’m writing a book called Surviving Complexity. Many of my posts here at this website have adapted from materials I’m writing for that book.

Well, now I’ve been hired by Ziff Davis Enterprises to write a weekly column on IT Management for the online version of Baseline. That column is titled “Surviving Complexity” and, again, draws upon work I’m doing for the book. My first column is up: “Lies, Damned Lies, and Metrics (Part I)”; here’s the opening paragraph:

When Capers Jones published Assessment and Control of Software Risks (Yourdon Press, 1994), he identified the most serious software risk in IT projects as “Inaccurate Metrics,” and the second most serious software risk as “Inadequate Measurement”. I remember being startled when I first read that back in 1995—they certainly weren’t what I would have chosen—and other authorities in the field criticized his choices. Yet, in the intervening years, I have moved closer and closer to Jones’ point of view.

I’ll still be writing here, both with materials relevant to Surviving Complexity and with my on-going revisions to The Art of ‘Ware. But please feel free to check out the new column. Thanks. ..bruce..

]]>
http://brucefwebster.com/2008/06/05/43/feed/
“Pitfalls of Modern Software Engineering”: an update http://brucefwebster.com/2008/05/29/pitfalls-of-modern-software-engineering-an-update/ http://brucefwebster.com/2008/05/29/pitfalls-of-modern-software-engineering-an-update/#comments Fri, 30 May 2008 02:35:49 +0000 bfwebster http://brucefwebster.com/?p=42 One of the books I’m currently writing is Pitfalls of Modern Software Engineering, a greatly expanded and updated version of a book I published back in the 1990s. I’ve been posted new and revised pitfalls over at my Bruce F. Webster & Associates (bfwa.com) website. To make the pitfalls a bit easier to browse, I’ve now added a page to that website that lists all the pitfalls posted to date, with link to their individual entries. Just FYI. ..bruce..

]]>
http://brucefwebster.com/2008/05/29/pitfalls-of-modern-software-engineering-an-update/feed/
The Arc of Engineering http://brucefwebster.com/2008/05/21/the-arc-of-engineering/ http://brucefwebster.com/2008/05/21/the-arc-of-engineering/#comments Thu, 22 May 2008 00:22:34 +0000 bfwebster http://brucefwebster.com/?p=41 [Copyright 2008 by Bruce F. Webster. All rights reserved. Adapted from Surviving Complexity (forthcoming).]

And so, from hour to hour, we ripe and ripe,
And then, from hour to hour, we rot and rot;
And thereby hangs a tale.

– William Shakespeare, As You Like It, Act II, Scene vii.

I have observed a pattern (or anti-pattern) in IT engineering that looks something like this:

Let’s call this the arc of engineering, since that’s more compact and elegant than the strangely shaped curve with what appears to be a single inflection point of engineering. “Arc” in any case conveys the essential sense: that system quality (however you wish to define that) rises over time to a peak value and then starts to decline.

This came to mind because I recently put a new hard drive in my laptop and have been going through the tedious drill of reinstalling all the various applications and utilities that I am used to using. With some software, I want — or have to use — the latest versions. So, for example, my versions of Firefox, Thunderbird, Quicken, and so on are all current, not to mention my anti-virus/firewall software.

For others, however, I either don’t need — or specifically don’t want — the newest versions. The laptop’s restore disks installed Windows XP, for which I’m grateful; I have no plans whatsoever to upgrade to Vista. Similarly, I installed MS Office 2003; I don’t own and don’t plan to buy Office 2007. As far as I can tell, Microsoft pretty much hit the top of its engineering arc for both Windows and Office a few years ago. In fact, the early to mid-2000s may well represent the peak of quality, features, and user acceptability for these product lines. Given the on-going end-user unhappiness with Vista and Office 2007 — and the corporate resistance to upgrades to either — Microsoft’s user base may never again be as happy with these product lines as they were a few years ago.

This is not an anti-Microsoft screed, because I have observed a similar effect in other companies and product lines. For example, I was a very early Macintosh adopter and champion, though I also was quite critical of its initial limitations. Apple continued to make improvements and changes to the Macintosh product line and finally reached what I considered at the time to be a close-to-perfect Mac: the Macintosh IIcx. It was solid, reliable, and very stable. Apple followed up with the Macintosh IIci, which used the same form factor, but offered several improvements over the IIcx, though there were some software stability problems due to the rewritten ROM and related hardware changes.

And then came the Macintosh IIfx, which was supposed to be even better — but the IIfx soon became well known for its hardware problems, in particular its tendency to blow out power supplies. Other Mac II models — the IIvi, IIsi, and IIvx — each had their own problems and limitations. From there, Apple descended into the product-line chaos known as Quadra/Centris/Performa, a marketing disaster from which Apple would not recover completely until Steve Jobs came back to Apple and introduced the iMac.

In this same time period, Apple had to deal with the fact that its operating system — System 7, introduced in 1991 — was under-featured, underwhelming, and built on aging technology. System 7 would lose its graphical UI edge with the introduction in 1992 of MS Windows 3.1 and would pretty much lose the operating systems wars completely a few years later with the introduction of MS Windows 95. It would not be until Apple’s introduction in 2002 of the Jaguar (10.2) version of Mac OS X — a heavily updated and retooled version of the NeXTstep OS — that Apple would regain technical and UI leadership in operating systems.

This same pattern can appear in internal IT projects. The system under development goes into production and continues to improve over time. Then at some point, the system’s overall quality (functionality, performance, reliability) starts to decline and maintenance costs begin to go up. At some point the organization decides to replace or re-engineer the system, sometimes successfully, sometimes not.

So, why does this happen? I believe several factors are or can be involved. (To simply the discussion, I’ll use “system” to mean the software, IT system, product, or product line under discussion; likewise, I’ll use “developer” to mean the company and/or IT development group responsible for creating, enhancing, and maintaining the system.)

The developer loses conceptual control of the system. In other words, no person or group of people understands the overall architecture, design, and operating constraints of the system. Thus the developer can no longer ensure that on-going changes to the system are consistent and compatible with existing aspects of the system or with other changes being made. This in turn can come about for several reasons:

  • The system grows too large and complex. Sheer code size is not necessarily a problem (though it doesn’t help); on the other hand, the degree of fragmentation and interdependencies can be. (See this Vista post-mortem for some examples.)
  • Those who really understand the system have left or moved on to other projects. The best and brightest IT engineers — particularly architects and designers — usually prefer to work on new systems. That means once this system has shipped (or gone into production), the people who created the system in the first place and who understand it best may well want to move on to other things, inside or outside of the company. Those who replace them — and those who stay behind — may be quite talented but may not understand all the system’s nuances and requirements. Eventually, they may decide to move on as well. With each generation of turnover, more institutional memory is lost.
  • The development organization no long meshes with the system. Conway’s law states that “any piece of software reflects the organizational structure that produced it.” Since the IT group responsible for the system often changes once the system ships or goes into production, the organization of the new group responsible for the system may not match the system’s structure, causing gaps in knowledge and responsibility, plus a tendency — conscious or not — to reshape the system to match the new organization.

Software rot sets in. Software rot is a real term of art within software engineering, referring to the well-known tendency of systems in production to become less reliable over time for a variety of reasons having mostly to do with piecemeal and uncoordinated changes to the system, or growing incompatibilities with external systems.

The enhanced system finally outgrows its original foundation. When the developer seeks to expand or improve the system, it often does so without properly rearchitecting and redesigning the system first. As a result, the original system eventually may be stressed beyond its original intent and begins to suffer from functionality, reliability and/or performance problems. Mike Parker, one of the co-founders of Pages Software Inc. (read the 3rd paragraph), once described this syndrome as “trying to build a castle on the foundation of an outhouse.”

Market or business needs shift beyond the product’s fundamental design. A developer creates and enhances a system to meet a specific set of business/market requirements. However, there often comes a time when new or expanded requirements exceed the capabilities of the system’s original architecture and design. This can lead to the “outgrowing the foundation” situation described above, with the resulting problems. Or it can lead to a true re-engineering or replacement effort, which may well lead to the final two problems below.

The developer begins to add “blue sky”/”kitchen sink” enhancements. Once the “door is open” on changes to the system — or features for the new/re-engineered system — the requests pour in from all directions. Without rigorous control over these enhancements, the development effort can quickly bloat in every aspect: architecture, design, code size, schedule, staff, and defects.

Backward compatibility is maintained at all costs. Supporting backwards compatibility in a system can be a bit like trying to drive a car while pedaling a wheel at the same time. Apple abandoned its old arc of engineering and created a new one under Mac OS X by first forcing binary backwards compatibility into a separate mode (the “Classic” environment) and then dropping the Classic environment altogether for Intel-based Macs running Mac OS X. Microsoft — which has struggled with the backwards compatibility issue for almost its entire OS life — has announced that for “Windows 7″, it will in effect follow Apple’s lead by not supporting binary (and to a certain extent source) backwards compatibility in Windows 7 but instead using virtualization technology (switching to a backwards-compatible OS) for older application.

I’m sure there are other reasons I haven’t listed here, but it does show why the arc of engineering exists. Comments? ..bruce..

]]>
http://brucefwebster.com/2008/05/21/the-arc-of-engineering/feed/
The engineering shortage: Japan http://brucefwebster.com/2008/05/17/the-engineering-shortage-japan/ http://brucefwebster.com/2008/05/17/the-engineering-shortage-japan/#comments Sat, 17 May 2008 12:39:38 +0000 bfwebster http://brucefwebster.com/?p=40 Today’s New York Times reports that Japan is “running out of engineers“:

After years of fretting over coming shortages, the country is actually facing a dwindling number of young people entering engineering and technology-related fields.

Universities call it “rikei banare,” or “flight from science.” The decline is growing so drastic that industry has begun advertising campaigns intended to make engineering look sexy and cool, and companies are slowly starting to import foreign workers, or sending jobs to where the engineers are, in Vietnam and India.

It was engineering prowess that lifted this nation from postwar defeat to economic superpower. But according to educators, executives and young Japanese themselves, the young here are behaving more like Americans: choosing better-paying fields like finance and medicine, or more purely creative careers, like the arts, rather than following their salaryman fathers into the unglamorous world of manufacturing.

The problem did not catch Japan by surprise. The first signs of declining interest among the young in science and engineering appeared almost two decades ago, after Japan reached first-world living standards, and in recent years there has been a steady decline in the number of science and engineering students. But only now are Japanese companies starting to feel the real pinch.

Read the whole article.  ..bruce w..

]]>
http://brucefwebster.com/2008/05/17/the-engineering-shortage-japan/feed/