Computer projects have failed for as long as there have been computers. But now that most companies are only as stable as their bits and bytes, the consequences of information technology screwups aren't easily disguised - they show up in earnings reports.
When IT goes bad, high-growth rocket ships like Oxford Health Plans Inc. and Ben & Jerry's Homemade Inc. report their first-ever financial losses. Others crater and run for bankruptcy protection, as did drug distribution giant FoxMeyer Corp.
In a Computerworld study of multimillion-dollar IT disasters, the following two not-so-surprising themes emerged:
• User companies like FoxMeyer often file tough-to-win lawsuits against the vendors or consultants involved. Nonetheless, collectively, users rarely seem to learn much from the episodes or apply the lessons to future projects.
• All of the botched projects in Computerworld's Top 10 disasters list were big and richly complex; many were the toughest IT projects the users had ever tried. Five were hideously difficult enterprise resources planning (ERP) system implementations.
Root Causes Remain the Same
The root causes of IT failures haven't changed a bit over the years.
 | | | Top 10 Failures See Top 10 Corporate Information Technology Failures in a PDF chart. (Requires Adobe Acrobat Reader) |
|
|||
Miscommunication, hazy goals, "scope creep," inept leadership, pitiful project management - you've heard, if not heeded, it all before.
"We may be neck-deep in the New Economy and Internet time, but you still have the same factors and the same failings," said Bruce Webster, a director at PricewaterhouseCoopers in Washington.
Webster recently studied 120 IT lawsuits filed since 1976, and he said he's convinced that most flops could be avoided if people simply knew the time-honored best practices of systems development.
| | | Dishonorable Mentions Some information technology disasters of the past decade aren't included in the Top 10 chart (at right) because their financial costs weren't clear or weren't quite as high as those that made the list. Others were quasigovernmental. But they were significant bungles nonetheless. Here's a sampling: • High hopes for a high-tech airport in Denver were dashed in 1994 when a baggage-sorting system misplaced and damaged countless suitcases. The object-oriented system, which was built by BAE Automated Systems Inc. in Carrollton, Texas, ran on IBM's OS/2. To fix it, primary sponsors Chicago-based United Air Lines Inc. and the city of Denver ended up paying at least $86 million more than the original $193 million price tag. The airport opened almost three years late. • Ben & Jerry's Homemade Inc., a small Vermont ice cream maker that hit the big time in the early 1990s, took a $4.1 million write-off in 1995 when it canceled a project to build a fully automated packing plant. As a result, the company posted its first quarterly loss ever. • Early this year, Thomas & Betts Corp., a $2.5 billion electrical parts maker in Memphis, blamed problems with a new Internet-based order-management system for a 50% drop in profits in the fourth quarter of last year. The homegrown mainframe system also cost the company another $42 million in order and shipping disruptions. In the following quarter, the company spent $11 million on customer service, extra freight costs and other measures to make up for ongoing system crashes and backlogs. In a lawsuit still pending, shareholders say Thomas & Betts misled them about the success of the new system. • Just in time for the Memorial Day holiday crush in 1998, Avis Inc.'s Wizcom reservation system crashed. The outage blocked the rental car company, as well as many travel agencies and hotels also linked to the system, from taking bookings for 30 hours. • In the summer of 1994, an error in a routine file update crashed the automated teller machine (ATM) system of Chemical Banking Corp. in New York. ATMs at Chemical, which was then one of the five largest banks in the U.S., were down for five hours. • In 1993, Fidelity Brokerage Services Inc. in Boston started a multimillion-dollar project to build a Windows-based trading application for customers with home PCs. "Jamaica" still wasn't ready by mid-1996, reportedly because of political clashes between quirky programmers and staid investment bankers. The application wasn't a failure; it worked when it was finally rolled out. But the numerous development delays put Fidelity far behind rivals such as Charles Schwab & Co. in San Francisco. |
| |||
"I don't know how many IT managers, team leaders, directors and CIOs have actually sat down and read The Mythical Man-Month, The Psychology of Computer Programming and Death March," he said, referring to three books that amount to the software development canon. "The causes of disasters are all well documented. They're fundamental."
Still, warning lights are easy to overlook when the whole room is spinning.
"There's a natural tendency to get overly committed to something, especially when there are no clear signals telling you you are off course," said Mark Keil, an associate professor at Georgia State University in Atlanta.
The infamously buggy baggage-handling system at the Denver International Airport is one case that offered unambiguous proof of technology glitches: shredded luggage.
But tests of most questionable IT projects don't yield such graphic evidence.
In large systems integration or ERP deals, "there's no torn suitcase sitting at your feet to wake you up," said Keil, who has studied IT disasters for nine years. "So it's a lot easier to delude yourself into thinking things aren't that bad."
Project Euthanasia
Euthanasia for the project might be the best course, but people often have too much heart and money invested to end it.
One technique for preventing a disaster is to add some humility to the endeavor. Invite a third party to review your work - a reliable consultant, an academic or a buddy CIO.
An outsider can "walk into the project setting for 20 minutes, talk to a few people and come to the conclusion that things have run amok. But people inside may not even be aware," Keil said.
Greyhound Lines Inc. in Dallas, for example, seemingly didn't know anything was wrong with its new reservations and logistics system - until it went live and 12% of its customers went away mad in one month.
Though specific individuals might learn from their own mistakes, those lessons aren't transferred to any collective IT consciousness.
"The people with that [failure] experience aren't always the people in authority the next time that situation arises," observed Kevin Hickey, a former head of IT at Trumbull, Conn.-based Oxford Health Plans Inc. "The fact is, hubris will always be with us."
And then there's what Webster calls "the thermocline of truth." Swimmers know that lake water separates into warm and cold horizontal bands. The area between is a thermocline.
In IT groups, everyone below Webster's "thermocline of truth" knows the project is sinking, while everyone above it thinks things are fine. Senior executives can be oblivious. They aren't involved enough, they don't want to have to face a failure, or underlings are afraid to tell them, he explained.
"You can see that persist almost until the point where the project is supposed to be delivered," he said. "Then, suddenly, it's, 'What do you mean this will take another six months?' "
That was part of the sorry plight of Fort Worth, Texas-based AMR Corp.'s Confirm reservations system. Confirm managers are even said to have orchestrated a cover-up.
Overall, IT culture is such that problems, especially expensive ones (which hold the most valuable lessons), are hidden. Programmers write around buggy code rather than tear it apart. Managers revise project specifications to reflect what they did instead of what they should have done. Senior IT leaders neglect to tell their bosses the bad news.
Most companies are too embarrassed to analyze their failures, said Effy Oz, an associate professor of management and IT at Pennsylvania State University in Great Valley.
"People will say, 'There's no time, and we're not paid to have these discussions,' " Oz said. "The CEO has to be a very confident person to say, 'These things happen. Let's learn from it.' "
The average loss in an abandoned project is $4.2 million, according to Oz. The blowups in Computerworld's top 10 list cost much more than that. And, if history is any indication, they will happen again.
Travel System Is Confirm-ed Disaster
On the heels of its hugely successful Sabre airline reservation system, Fort Worth, Texas-based AMR Corp., the parent company of American Airlines Inc., in the late 1980s formed a joint venture with Marriott International Inc., Hilton Hotels Corp. and Budget Rent A Car Corp. to build a similar system for the travel industry. But Confirm, as the project was called, wasn't to be. In fact, the effort is viewed as one of the worst IT failures ever for its mismanagement, questionable ethics and unworkable software.
AMR's information systems unit in Dallas was the lead developer on Confirm, which was originally due in June 1992 for no more than $55.7 million. Yet Confirm started to miss deadlines and cost estimates several months after development began.
Specifications were unclear, unskilled programmers were assigned to the project and mainframe-based transaction-processing software couldn't be integrated with a related mainframe decision-support system. Moreover, one year into design and development, Confirm had already fallen one year behind schedule.
Bethesda, Md.-based Marriott and Lisle, Ill.-based Budget started asking questions in 1990 but were assured that Confirm would work and that programmers would make up time and meet the deadline.
In April 1992, just three months before it was slated to go live, Confirm failed tests at Los Angeles-based Hilton. AMR also told its partners in a letter that it needed another 15 to 18 months.
"The individuals whom we gave responsibility for managing Confirm have proven to be inept [and] concealed a number of important technical and performance problems," the AMR letter said.
The legendary Max Hopper, who had been instrumental in Sabre's development, was vice chairman of AMR's IT unit at the time, though not directly involved in the daily work of Confirm. However, he acknowledged in a note to his employees that some Confirm managers "did not disclose the true status of the project in a timely manner. This has created more difficult problems - of both ethics and finance - than would have existed if those people had come forward with accurate information" [News, May 22, 1995]. After consuming almost four years and $125 million, Confirm was effectively dead.
In September 1992, AMR sued Budget, Hilton and Marriott; Marriott then sued AMR. The suits were settled out of court for undisclosed terms. Hopper recently declined to discuss Confirm, citing the secret settlements.
AMR was mainly to blame, but all sides acted unprofessionally, said Effy Oz, an associate professor of management and IT at Pennsylvania State University.
Executives at AMR "simply lied to their client-partners," said Oz, who has studied Confirm. "The partners were irresponsible in not asking more questions more often and in accepting at face value all that [AMR] told them."
Budget, Hilton and Marriott today use their own - separate - reservation systems.
A Really Bad Bet for Drug Distributor
When it launched a $35 million enterprise resource planning (ERP) project way in 1993, FoxMeyer Corp. was a $5 billion drug distributor in Carrollton, Texas.
Now it's bankrupt.
It wasn't the fumbled IT project alone that destroyed FoxMeyer, but that was a critical contributor, according to lawsuits FoxMeyer filed against SAP AG, SAP America Inc. and Chicago-based Andersen Consulting in 1998.
SAP lied repeatedly about R/3's capabilities, and Andersen's inexperienced staff used FoxMeyer as a training camp, claims the drug company, which seeks damages of $500 million from SAP and $500 million from Andersen.
FoxMeyer was one of the first big users to take a chance on ERP, which was a hot new idea at the time. Perhaps the company should have been more cautious. Legal documents show that FoxMeyer knew that SAP's R/3 software hadn't yet been used at distribution companies - just at manufacturers.
Still, FoxMeyer was jazzed. Then-CIO Robert Brown told Computerworld in 1994, "We are betting our company on this."
Big problems started to emerge later that year. For example, the new R/3 software miscounted inventory, which in turn screwed up customer orders. Outright crashes were routine.
SAP declined to talk specifically about FoxMeyer. But an SAP spokesman said users who install R/3 are usually changing basic business processes at the same time, which "is typically where most of the pain and challenges of implementation come from."
FoxMeyer also charges that R/3 performed worse than the company's proprietary Unisys Corp. system. R/3 could process just 10,000 invoice lines per night, compared with 420,000 for the Unisys setup.
SAP misled FoxMeyer with faulty benchmarks, according to the suit. Other users have also questioned SAP's benchmarks [News, Sept. 4, 1995]. SAP doesn't misrepresent benchmarks, the company's spokesman said.