The Unhappy Valley: customized COTS between “buy” and “build”

In response to my post yesterday on buy vs. build, Geoff asked a very important question:

I’d be interested to get your opinion on the degree of customisation that COTS packages can realistically handle. I’ve seen several systems that started out as customised COTS, but eventually departed so far from the vanilla functionality that we were writing “business rules” that simply bypassed the built-in processes and called out to a custom java function. At this point, you’ve got all the downsides of creating and maintaining your own code, *plus* the hassle of fighting against the package framework, reworking the customisation every time a new version is released, etc – clearly not optimal. The sad thing is that this isn’t rare: the alternative is to change the business to fit the package, which might be a great idea, but it’s much more difficult to persuade the business to change than to sign-off a huge, doomed software integration programme. So in your experience, how much customisation is too much?

Let’s start first by re-examining the two ends of the spectrum. At one end, you have a pure commercial-off-the-shelf (COTS) software package that you license, deploy, and use. There may be some degree of configuration involved, but no customization by the vendor on your behalf. If you have XSPOT 4.3, and the vendor releases XSPOT 5.0, you are generally free to make the decision whether to upgrade to 5.0 or keep using 4.3.

At the other end, you have a pure custom system, build from scratch by your own (or hired) developers. You own all the source code, you have a development lifecycle in place so that you can add features and fix bugs as desired, and you can make it jump through whatever hoops you are able to make it jump through.

Now, to a large extent, both ends are a myth. For anything beyond a relatively simple application, a given COTS package will come with its own baggage: operating system and hardware platform requirements, dependency upon other COTS products, and an eventual end to vendor support, either through an explicit end-of-life declaration for a given version or for the entire package (cf. Windows XP facing end-of-life in 2014), or because the vendor itself is bought up, goes out of business, etc.

Similarly, pure custom software will likely have its own dependencies on commercial products (programming language versions and libraries, operating systems, other custom and off-the-shelf software packages, etc.). Beyond that, software entropy starts to set in; improvements and bug fixes are increasingly done by maintenance personnel who are more interesting in closing out a feature request/defect report than preserving architectural and design consistency, and you end up with something like this:

So you have some real issues you have to deal with from the very start even with a “pure” buy or build solution.

Now, having laid all that groundwork, we can enter into the Unhappy Valley represented by an increasing amount of customization of a COTS package.

First, for customization of a COTS package to happen, you have to be paying enough money for the vendor to either do the customization for you or to give you the necessary portions of source code to customize yourself. So the first-order analysis is to ask yourself whether this may end up costing more and/or taking longer than doing the software yourself. The immediate answer is likely no, but you need to think long and hard about two consequences: maintenance costs and forked versions.

Let’s say you want to license XSPOT 4.3, but you need it customized. You figure out all the changes required, and you and the vendor, XCorp, reach an agreement for XCorp to do the customization. After some period of time, they deliver to you XSPOT 4.3U, your custom version. But along the way, XCorp announces XSPOT 5.0, which includes significant performance and feature improvements and a major user interface redesign. You like to have a customized version of XSPOT 5.0, but XCorp tell you that because of the refactoring and redesign for XSPOT 5.0, your customization work will largely have to be done all over again and will cost pretty much what you’ve already paid for XSPOT 4.3U. So you pony up again, and at some point after the 5.0 release, you get the 5.0U version.

As time goes on, XCorp issues versions 5.1, 5.2 and then 5.3 of XSPOT. You have a (hefty) maintenance agreement with XCorp, so with some delay, you receive and deploy in your organization 5.1U, 5.2U, and 5.3U.  But then your industry has some very significant changes, and you need some extensive modifications to XSPOT to keep up with the industry. XCorp agrees to make those changes — again for a very healthy sum — but warns you that they will preclude you transitioning to XSPOT 6.0, which is now under development.  So XCorp creates 5.5V for you and agrees to support it for two years (for a hefty maintenance fee), but you are not going to get the improvements and bug fixes that will go into XSPOT 6.0 and later.

At this point, your version of XSPOT has forked from the mainstream version of XSPOT, and your finding yourself paying a pretty penny for bug fixes and minor feature upgrades. But after a while, you start running into new problems. Other firms with which you interact electronically have shifted to XSPOT 6.0 (and beyond), and you start running into compatibility issues between your systems and theirs. Now you start investing in custom bridgeware to ensure that your systems can speak with theirs.

As the two years expire, XCorp states that it no longer has the resources to dedicate to XSPOT 5.5V, but for a very large fee and under some strict intellectual property protection agreements, you can license the 5.5V source code base and support the package on your own. Suddenly, you’re in a situation roughly equivalent to having a custom software package, except you have absolutely no in-house technical expertise with regards to 5.5V’s architecture, design, or source code. If you’re lucky, you can find a former XCorp programmer who could help support the package, but if she’s any good, she probably doesn’t want to take what is likely a dead-end job supporting a custom software package with no future.

That’s a long and painful way of answering Geoff’s question like this: any amount of customization that keeps you from getting all future updates of the COTS product in a reasonable time and for a reasonable fee is probably too much. Otherwise, you’re going to slowly descend into that Unhappy Valley, and getting out is going to be lot harder — or more expensive — than you think.  ..bruce..

About the Author: bfwebster

Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com.