Weighing in on Project Orca

[Cross posted from And Still I Persist]

[Note: I am currently in transit from Colorado to Florida and am composing this post as I have time and ‘net access.]

“All the most important mistakes are made on the first day.”

– The Art of Systems Architecting (Maier & Rechtin)

Project Orca was the Romney campaign’s technological effort to track in near-real-time actual voting in precincts all over the United States, with the intent of using that information in combination with individual-specific demographic databases to increase or pull back phone-call and door-knock get-out-the-vote (GOTV) efforts based on what was happening there. Since it appears more and more that Romney lost largely because of a lack of Republican turnout, it’s safe to say that the Romney/RNC GOTV effort was a failure. What is less clear is how much of an impact Orca had on that failure.

Three important disclosures up front. First, I was a Project Orca volunteer in Colorado. Second, I have no direct knowledge of how Orca was envisioned, developed, and operated beyond my own experience as an end user. Third, I’ve worked in software engineering and information technology since 1974, and my major professional focus since 1994 or so has been on how and why IT projects fail or succeed — a subject on which I’ve taught seminars, published books and articles, consulted in large corporations, and testified in courts and before Congress.

That said, the flap over Orca has made it all the way to the Drudge Report, largely due to two key discussions. The first is by JohnE over at Ace of Spades, who wrote the initial scathing report on Orca and followed it up with a reply to some of the push-back from the Romney campaign. The second is by Joel Pollack over at Breitbart, who had a Romney worker in Colorado who spoke of some of the problems.

My own experience matches much of what JohnE (who was also in Colorado) stated. I participated in four training calls. The first three all gave the same information about how the system would work, though — as JohnE noted — the Romney people kept refering to the Orca software as an “app”, even though it was simply a website that you logged into. (From JohnE’s comments, it sounds as though the use of the term “app” confused at least some Orca volunteers.)

Unlike JohnE, I was very clear from both the calls and the e-mails I receive that I needed to pick up my poll watcher certification and bring it with me to my assigned polling place on Election Day, so I am curious about his confusion. On the other hand, for my particular county (Douglas County), it appears that the certifications weren’t ready until the day before Election Day, which strikes me as cutting it a bit close. But I did drive down to Castle Rock on Monday afternoon and picked it up.

There was one more call the night before Election Day, but that was just a very brief, rah-rah call. After that Monday night call, I tried logging into the Orca web site to test out my password, but the log-in failed. I found that puzzling, but just assumed that they had the system locked down ether for security or administrative purposes.

Since Colorado is one of the few states that does not allow any electronic devices (phones, tablets, laptops) in polling places, I was also very clear from the training that I would have to print out the list of registered voters, bring it with me, mark off voters as they identified themselves at the polling place, and then periodically run out to my car and enter voting information via my iPad or iPhone. I printed out the list — and immediately discovered what I considered the first, admittedly minor, glitch: the list, a PDF file over 60 pages long, was formatted in such a way that you could not three-hole-punch the sheets (to put in a binder) without punching through the check boxes (for tracking voting and reporting) for some of the voters on each page. As I said, minor, but my technical writing background made me frown a bit. Since I own Adobe Acrobat, I solved the problem via the kludge of adding headers and footers to each page and then telling the pages to shrink themselves to fit.

So, with two binders in hand (I printed two copies of the list), with iPhone, my iPad, and my laptop in the car — along with two Black & Decker power boxes, and a couple of folding camp chairs — I showed up at my polling place at 6:45 am. I also brought with me three dozen fresh donuts (a suggestion from one of the training calls) that were greatly welcomed and did much to smooth relations with the actual poll workers.

Everything went well until the first time I went out to the car to report voters. I tried to log into the Orca website, and once again I got a “login failed” error message. (Second minor but telling item: that error message had a typo in it. The fact that such a typo was in a high-probability error message in a live system with over 30,000 anticipated users speaks to haste and sloppiness.) I called the help number given in the error message, and got a support person at the Boston command center who said they would reset my password and gave me the new code, but that it would take a little while for the new code to go live.

When I came back out again an hour or so later and tried to log in, the new password didn’t work either. The Orca system also had a call-in option, so I tried that and was told that my PIN was invalid.

At this point, it was about 8:30 in the morning. The next eight or so hours were all more of the same — I would call, the password would be reset, I would wait a while, I would try to log in, it would fail. Rinse and repeat. I was told by some of the support people I spoke with that almost no one in Colorado was able to log in — a geographical uniqueness that I found puzzling, since the login screen just asked for username (an e-mail address) and password. Another support person mentioned that there were similar problems being reported for North Carolina. Finally, sometime around 4:30 to 5:00 pm, I was given a new pin for the dial-in line — and that finally worked. I reported all the voters who had voted so far that day and did updated reports twice more before the poll finally closed at 7:00 pm.

JohnE pushes back (with justifiable skepticism) against Zac Moffat’s claims that “the campaign had voting data from 91 percent of counties” — not that they is necessarily inaccurate, but that it could be largely irrelevant. As JohnE points out, there can be hundreds of precincts in a single county; beyond that, having voting data from a given county does not necessarily mean having actionable, timely, and sufficiently complete data throughout the day in order to make GOTV decisions for that county (or for a given precinct in that county).

There is still no evidence that there were significant Orca problems outside of Colorado; we’ll have to wait to see if someone on the inside of Orca development decides to leak. But here are some observations from an large-scale IT system development perspective.

“A complex system that works is invariably found to have evolved from a simple system that works. . . . A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.”

— Systemantics (John Gall)

First, the Orca folks basically released version 1.0 (or, perhaps more accurately, version 0.9 or lower) into a massive, live environment on the single most critical day, Election Day. This is commonly known in IT circles as the “Big Bang” approach, and it is a well-known harbinger of disaster. Even if the Orca developers were all geniuses, this still would be a grave error. We’re not talking about an iOS or Android app being used by a single person on his/her device for entertainment or productivity; we’re talking about a massive web-based system that has to accept data from (not just deliver content to) possibly tens of thousands of users simultaneously and in real time. There should have been days, if not weeks, of live tests and trials to see how the system performed and to get the kinks worked out. Instead, as far as I can tell, the first time most Orca volunteers actually used (or tried to use) the system was on Election Day. Even if the system worked perfectly, I doubt that the majority of users — who had no real training on the system — would have used it in an effective manner.

“There are products that you shouldn’t develop, companies you shouldn’t challenge, customers you shouldn’t win, markets you shouldn’t enter, recommendations from the board of directors you shouldn’t follow.”

— The Art of ‘Ware (Webster)

Beyond that — and as per the quote above (paraphrasing Sun Tzu), as well as the one at the start of this post — the real question is: should this have been attempted in the first place? There are anecdotal reports that the focus on Orca took away resources — especially feet-on-the-ground resources — for local GOTV efforts. And GOTV is exactly where the Republicans failed. If work on Orca had been started 3 or 4 years ago, and if it had been used on Election Day 2010 — both as a real-world test and to verify the expected benefits of matching actual voters with demographic information to predict results in a given precinct — then it might have been ready for prime time on Election Day 2012.

But to spend signficant time, money, and resources (including tying up some 30,000+ local volunteers) for an untested system and an untried concept was a recipe for disaster — a well-known recipe, I might add, for those of us who have studied large-scale IT projects (which, frankly, are prone to failure anyway).

“But,” you ask, “you signed up to volunteer.” Yes, I did. But I had no visibility into the project and no inherent reason in advance to assume that things weren’t being done well, beyond the few niggling questions I had prior to Election Day. And, to be perfectly fair, I have no evidence of the extent of problems beyond Colorado, aside from the acknowledged 90-minute downtime on Election Day morning. But, even if those are the only two failures, can you seriously argue that it was a success? A time-critical, mission-critical system whose entire purpose for existence is to be used in a single 12-hour window is down for over 10% of that time in all states and is down all day in one critical swing state?

And, finally, the bottom line: if the purpose of Orca was to provide real-time intelligence on actual voting trends and make GOTV more effective, then where are the results? By all accounts, the Romney campaigned was stunned by their loss; if Orca was providing accurate, real-time indication of voting, as the project directors claimed it would in the training calls (actual quote [from memory]: “We’ll know ahead of anyone else how the election is turning out.”), then Romney should have had his concession speech written well in advance. And if its goal was to trigger targeted GOTV efforts in swing states, then why did Romney lose almost every swing state? Measured by its stated goals, Orca was a failure, pure and simple. Worse yet, it may have diverted critical resources from more effective GOTV efforts.

“Start out stupid and work up from there.”
— Bruce Henderson

I’d love to be able to do a project post mortem on Orca, as I have done on so many troubled, failing, or failed projects. But even without access to people and documents, I can hazard a guess at some of the likely factors:

• A late start and an impossibly short schedule.
• Because of these above, a waterfall (single pass) approach to design, development and deployment, instead of an iterative approach.
• Lack of a single chief software architect, with resulting problems at subsystem interfaces.
• A large development team with no prior track record developing and releasing software as a single team.
• Along the same lines, key components divided up among separate teams with relatively late integration and testing.
• And possibly some turnover (departure) of key personnel through the project, along with new personnel added late.
• Insufficient resources (money, personnel, equipment, time) devoted to quality assurance — and by “quality assurance”, I mean not just a wide range of testing (including, as JohnE points out, stress testing), but also individuals and teams with proper expertise, agreed-upon standards and guidelines, reviews, metrics, defect management, and so on.
• A firm belief — without any real-world evidence — that the idea would actually work and that the results would actually be worth the diversion of resources (including volunteers).

I’m happy to talk with anyone from the project who wants to set me straight or explain what really happened. But, in the end, what really happened was that Orca failed to achieve its goals, and Romney lost what should have been a very winnable election. ..bruce w..

About the Author: bfwebster

Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com.