Distributed Development (Part I)

The idea of an IT project team spread out over a wide geographical area is an old one. It became practical in the late ’70s and early ’80s, with the advent of hard disk drives — instead of punched cards, paper tape or magnetic tape — for source-code file storage, wide-area networking (including the Internet), modems, and personal workstations and computers. Early logistical problems, such as source-code control and configuration management, were largely solved or at least brought under control with a rapidly growing and evolving set of tools. That technology has continued to advance, particularly with the advent of a large number of Web-based tools to allow collaborative development anywhere an Internet connection and a browser are available.

In the years since then, organizations of all sizes and purposes have sought to develop software in a distributed fashion. The motivations have varied. Some have sought to coordinate the work of IT development shops in different corporate or governmental sites to bring more IT engineers to bear on a given project. Others have seen it as a telecommuting or quality-of-life issue, allowing developers to work from home. Yet others have sought to control IT costs by moving entire projects or portions thereof offshore to India, China and elsewhere.

With consistently high oil prices (ramped-up US production notwithstanding) and the corresponding high level in all transportation costs, the increased financial burden of getting engineers together in the same location, whether it’s driving to work or flying across the country, makes distributed development all the more attractive.

Whatever the motivation, the real problems with distributed development — which persist to this day — are the human ones, such as communications among engineers, conceptual unity of the system under development and personnel management.

It is a long-standing axiom in the software engineering world that the best software is written by a small number of talented developers working in proximity to one another. Why?

• They can easily interact and solve problems as a group.
• Each developer can readily ensure that his or her work meshes well with everyone else’s and with the system as a whole.
• Everyone knows what everyone else is working on.

As you increase the number of IT engineers on a project, the communications issues increase at a much greater rate. There are more possible communications pairs (for the mathematically inclined: n(n-1)/2 pairs for n engineers). The physical distance between engineers of necessity increases (any two engineers are now sitting farther apart), reducing the chances for interactions. And it becomes harder for each engineer to keep track of and coordinate with what all the other engineers are working on.

On top of or in place of this, imagine actual dislocation: engineers on the same project too far apart to walk over and see each other — whether they’re in different buildings, different cities or different countries. Add to that significant differences in time zones, cultures or even languages, and you can began to see why such efforts often fail.

Yes, the engineers can pick up the phone, chat via instant messaging or send e-mail. But any software engineer will tell you that’s not the same as impromptu sessions over a cubicle wall, in each other’s office, in the hallway, at a whiteboard, or while eating pizza together late at night — in each case with other engineers joining in as they hear what’s being hashed out. One senior engineer I know spent two of her four years on a large software project in a different state from the rest of the development team. She talked about how much out of the loop she felt during that time and how limited she felt in her ability to make contributions to technical issues in the project.

This communication issue leads to a lack of conceptual unity within the software itself. Fred Brooks famously talked about the need for conceptual unity in large software projects and, thus, the need for a chief architect. However, as Tom Affinito once told me, architecture is a political act that requires constant interaction with other team members to increase understanding, gain buy-in, and establish consensus regarding the foundational principles and aspects of the system.

Such architectural politicking is hard enough with a development team all in one location. When the team is distributed, it can become very difficult. As a result, the developers began to drift from one another in their understanding of how the system is intended to work and to fit together. You can end up with subsystems that may both adhere to a documented API but that have some deep internal mismatches on what leads up to or happens after a given API call.

I remember one project for which I was chief architect where we had two groups of developers, one on the West Coast (where I was) and one on the East Coast. I wrote plenty of architecture documents, sent e-mail and made phone calls to those on the East Coast, and we even had some videoconferences. But I found it much harder to bring the East Coast developers in line with what we were going to develop; the bandwidth just wasn’t there. To make matters worse, we were on a very tight schedule (despite my strenuous objections).

Finally, there’s the down-to-earth issue of management.

If you’re the manager for a distributed IT project, how do you manage your personnel successfully? How do you know what they’re doing and how they feel about it? I’ve managed groups of developers on large, long-term projects; you need to be able to walk around and talk to each developer regularly and casually.

Again, picking up the phone or sending e-mail is not the same as a quiet (or not-so-quiet) heart-to-heart in a private office or conference room.

Good team management can make a significant difference in a project’s success or failure, and it’s not as though IT engineers are the easiest group in the world to manage (the joke about “herding cats” was around for at least a decade or two before someone made a commercial about it).

Distributed development makes management just that much harder.

In short, distributed development is hard, and a lot of projects founder because of those difficulties. In the second part of this post, I’ll talk about some successful real-world techniques of distributed development that may help you make it work in your own projects.

[This is adapted from an article I originally wrote for the online version of Baseline.]

About the Author: bfwebster

Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com.