Pulling the Plug

War stories -- and lessons learned -- from IT leaders faced with products that didn't work as advertised.

Your organization has invested thousands, maybe millions, of dollars trying to install a vendor's product to give your Web site more capacity or optimize your supply chain, and it just doesn't work as expected. Now what?

Senior management wants to know where the money went. End users want to string you up. You'd like to sneak out of Dodge, but you can't.

Here are the tales of six IT leaders who had to deal with underperforming projects and took decisive actions to clean up the mess.


Four years ago, the U.S. Air Force tried to replace a 25-year-old aircraft maintenance tracking system with one that had mobile terminals and Unix-based off-the-shelf software that it expected to begin piloting within 12 months.

But after two years and heavy customization of the commercial package, "we realized it just wasn't going to happen," says Air Force CIO John Gilligan.

The Air Force tracks many of the parts used in its aircraft by their serial numbers, and the third-party software it was trying to deploy "couldn't do the kind of serial-number tracking we needed," Gilligan says.

In addition, he says, the software "wasn't mature enough" to handle the Air Force's stringent quality-assurance sign-offs on aircraft maintenance.

In hindsight, Gilligan says the Air Force was partly to blame for what became a $200 million failure -- a price tag that includes software licensing and consulting fees and the cost of hardware and internal labor. He says that the Air Force did a poor job of governing the project and that critical decision-making during the project occurred on "too low a level."

"It was kind of a slippery slope, and we were so far down that it became very difficult to walk back up," says Gilligan.

When the agency decided to cancel the project, end users "were doubly unhappy," says Gilligan. Not only would the system not be delivered, but the Air Force also froze any upgrades to the mainframe system for four years to help free up capital for the modernized system.

So Gilligan and the Air Force team addressed some short-term requirements to improve the end-user experience. Two years ago, the agency developed a browser-based front end to make it easier for maintenance crews to access the legacy system from its 100-plus worldwide bases.

The project team also began collapsing dozens of databases into a single repository. Then the group began to implement middleware tools and portal interfaces to make it easier for top brass to pull maintenance information out of the system.

Cutting the cord with the vendor behind the project wasn't too difficult, says Gilligan, since the contract had various milestones that had to be met or the effort would stop. And while the vendor "realized they would get a black mark" for not having completed the project, "on some level, they were just as unhappy" with the Air Force's inability to re-engineer its internal processes, Gilligan says.

Now that the commercial market for maintenance systems has matured a bit and the life of the legacy system has been stretched out, Gilligan says his group will begin evaluating other third-party packages within the next year, including ERP systems that would include components such as supply chain management, maintenance and procurement functionality.

Less Than Hospitable

At Mandalay Resort Group, marketing managers "were livid" about an effort to integrate two marketing applications, a project that ran two years late and 25% to 30% over budget, says CIO Tracy Austin. The marketing data warehouse encountered systems throughput and uptime problems related to the hardware, business intelligence software and the operating system that were selected, says Austin.

Part of the problem, she says, was that the Las Vegas-based hospitality and gaming company had failed to set any performance parameters to measure at regular intervals whether the project was meeting business and technical requirements.

Austin, who joined the company as CIO in early 2003, when the project was at its midpoint, took immediate steps to salvage the seven-figure effort. With no contingency plan in place, Austin went to senior executives in the company and explained the situation to gain additional funding for new hardware and software. She made sure the contracts with the new vendors included financial penalties if systems availability and throughput thresholds weren't met.

Mandalay then installed a new operating system and hardware platform within two months. Once that "healed the bleeding," the project team installed new business intelligence tools six months later, Austin says.

Because the previous system had been in place for nearly two years and was being used by Mandalay's marketing team, says Austin, "there was no way to recoup any significant portion of the investment." But it did help teach her staff the value of establishing strict prototype and design reviews and building performance metrics into the contract.

Authorizing Success

In mid-1998, MasterCard International Inc. began looking for a client/server authorization system for its merchant customers as part of a five-year, $160 million IT upgrade. But 18 months into what was supposed to be a one-year effort, the Purchase, N.Y.-based company realized that the number of lines of code it had co-written with the vendor had doubled and that the system still had big problems porting between HP-UX and Sun Solaris.

"There was a lot more vaporware there than software," says Robert Reeg, senior vice president of systems. Fortunately, MasterCard had built termination clauses into the contract as well as intellectual property protection. As a result, the credit card issuer was able to obtain full ownership of all the software it co-wrote with the vendor and free licensing rights to the base software code. MasterCard finished developing the software itself and put it into production in 2000.

Although that piece of the company's five-year systems enhancement effort ran about 25% over budget, the overall project was completed on time and just 3.7% over budget, Reeg says.

Reeg says he learned a few lessons from the experience, including the importance of running a vendor's software through a third-party lab to ensure that it meets performance requirements. And "having a good lawyer working with you" to help define exit clauses and intellectual property rights in the contract doesn't hurt either, he adds.


Sometimes, pulling the plug on a project can be a blessing for everyone involved. When Juniper Networks Inc. was about six months into developing a browser-based, self-service customer system in late 2002, IT staffers and customer service managers realized that the system they had licensed just wasn't meeting the company's needs, says CIO Kim Perdikou.

The business-unit leader agreed, so Perdikou negotiated a contract termination with the vendor and had internal staffers develop the desired functionality in-house.

"The biggest thing was the relief people felt when ... we tried to do something totally different," she says. "People were stressed about this thing not working properly."

The Sunnyvale, Calif.-based router maker had a 90-day limit on all projects, but the standard wasn't enforced as strictly then as it is now, says Perdikou. The project ended up running 90 days longer than expected and 50% over budget.

But the biggest takeaway, says Perdikou, was learning to admit openly to end users that the system wasn't working properly and to tackle the problem head-on.

"The expectation of IT delivering what you need today is so low that users give up and build what they want or live with things that are semi-best," she says. "We should work with people to get things done right."

Having a solid contingency plan can also ease end-user tensions. When Philadelphia-based Lincoln Financial Group tried to install an enterprise single sign-on customer identification system for its external Web sites in late 2003, the product couldn't perform the kind of distributed administration that the IT department or end users had anticipated, says Jason Glazier, Lincoln's chief technology officer.

So Glazier had the project team take out the vendor's product, and Lincoln developed its own identification system using Microsoft Corp.'s SQL Server. That helped the project team meet its original delivery target of January 2004.

And although the project sponsor was dismayed that the original product didn't work as expected and probably cost an extra $50,000 to $100,000, having a bona fide backup plan "made it a much less awkward" situation, Glazier says.

Copyright © 2004 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon