After the fact: How to find out if your IT project made the grade

A 'postgame' project analysis by key players is worth the effort.

Ask most IT managers why their organizations don't regularly conduct postmortem reviews on completed IT projects, and the typical response is, "We'd like to, but we just don't have the time or resources."

But that hasn't stopped organizations such as the Chicago Mercantile Exchange and Solo Cup Co. from regularly reviewing at least a portion of their completed IT projects. And while some of the reviews are done to see if a project met its anticipated objectives, many postmortems are conducted simply to determine what the project teams could have done better.

In this age of austerity, either objective can be a win for IT as postmortems prove the business value of projects or boost continuous improvement efforts.

"Each year, you want to do better than the previous year" in terms of project planning and project portfolio management, says Carl Stumpf, director of project and financial controls for the technology division at Chicago Mercantile Exchange Inc. (CME).

But Stumpf is in the minority, according to Gartner Inc. Just 13% of Gartner's clients conduct such reviews, says Joseph Stage, a consultant at the Stamford, Conn.-based firm.

There are a variety of reasons why postmortems aren't conducted, and some of them are defensive. "If the projects didn't accomplish what you set out to do, no one wants to go back and disclose that," particularly if such discoveries end up negatively impacting a project manager's performance review, says Tom Bugnitz, a consultant at Arlington, Mass.-based Cutter Consortium and a partner at The Beta Group, a St. Louis consulting firm.

Ironically, the IT groups that need postmortems the most are probably the least likely to perform them. The notion of conducting a postmortem "would be viewed as a time sink" for an IT organization with immature project management capabilities, since it wouldn't be able to fully grasp the value to be gained, says Margo Visitacion, an analyst at Cambridge, Mass.-based Forrester Research Inc.

Seeking Returns

But postmortems can pay off for companies that expend the time and resources to execute them properly.

When the CME, a predominantly electronic futures exchange, went public in 2002, that placed "a lot of the risk on our side" in IT, says Mark Bennett, associate director of the project and financial controls group. To help mitigate some of that IT risk, the CME began conducting postmortem reviews on its two-dozen largest IT projects in late 2003.

The exchange uses enterprise portfolio management software from Newport Beach, Calif.-based Artemis International Solutions Corp. to help project teams go back and evaluate their original objectives, risks and assumptions and determine whether projections for resources, capital and contractor fees were estimated accurately, says Bennett. Those steps enable the CME's project sponsors to better understand the financial constraints the technology division's project group faces and help them to better balance risk and prioritize projects.

The reviews have also helped the CME's project teams determine how to improve future iterations of software development projects, says Peter Barker, director of interest-rate products for the exchange and a business sponsor of IT projects.

He points to an enhanced options system that went live last August that enables the exchange's market makers to trade euro/dollar options on computer screens. One of the "thousand little things" that have come out of that project's review were recommendations made by market makers to improve an application programming interface that allows them to send "streaming" prices to trading screens, says Barker. Those improvements were put into production in February, he adds.

But sometimes it takes a while to get the postmortem process right. Meijer Inc., a Grand Rapids, Mich.-based chain of grocery stores, began conducting postmortem reviews on some of its IT projects after it formally launched a program management office (PMO) seven years ago, says IT program manager Jim Morse. But the practice was abandoned after a few years when the review process became "too picky," he recalls.

"We were trying to do metrics and measures and turn it into a scorecard, and it got kind of scary," says Morse. That's because the review process at that time was focused too much on metrics and failed to evaluate some of the softer but equally important aspects of project success that can't easily be measured, such as trust, commitment and reliability, he says.

The project reviews have since been restarted and are now focused on more-qualitative measures: what went well, what didn't go well, what could have been done better and what steps could have been taken to improve a project's quality.

Meijer's PMO conducts postmortems on only some of its IT projects, since "we just don't have the bandwidth" to examine all of them, says Morse. Sometimes they're done at the behest of senior management to determine whether a big project met its business objectives. Morse will also occasionally conduct a review of a project in progress if he thinks it's running off track.

No Scapegoats

One of the big challenges in postmortems is to convince project managers that they aren't being made scapegoats for problems that might have arisen.

"The first reaction is, 'Oh, my project is a failure and now they want to nail me,'" says Morse. "But it's not an 'I gotcha' game." Used properly, postmortems are an opportunity to review the strengths and weaknesses of how a project was handled and learn from them.

IT project teams at Solo Cup started doing postmortem reviews two years ago on roughly 20% of the company's IT projects. The reviews give IT a "grass-roots perspective" on problems that have cropped up and help to determine what could have been done better, says Richard Wolfson, senior manager of IT quality assurance at the Highland Park, Ill.-based maker of disposable tableware.

At Solo Cup, the quality assurance person who is assigned to the project runs the review, says Wolfson. Most of the reviews examine technical issues: Was project planning done effectively, and was it well coordinated with business users? Did testing follow the proper procedures? Were system defects caught quickly during the development phase and resolved?

If there are only technical issues to resolve, business sponsors are typically left out of the review, says Wolfson. But in about a quarter of the cases, their input is expected to help, so discussions with the business sponsors are included as part of the process.

The CME's Stumpf says that aside from helping to mitigate IT risk and improving IT product iterations, his reviews have yielded additional benefits, including increased visibility for the IT department. "There's a subtle benefit to it," he says. Business managers buy into IT projects much more, because postmortems give business sponsors a clearer picture of what the IT project group is doing.

Postmortems can also bring unexpected benefits. Last year, Meijer's PMO reviewed a big replenishment system project. The review included interviews with the vendors involved. And although its results weren't particularly surprising, it did lead to changes in relationships with some of the vendors, says Morse.

But perhaps the most important benefit is one that occurs before the postmortem even begins, says The Beta Group's Bugnitz: "It forces people to place greater attention on achieving the desired benefits at the front end of the project."

Copyright © 2005 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon