The Fuzzier Side of ROI

Don't be fooled by CIOs who claim that every IT project needs to pass a rigorous ROI test. They don't really believe it. They're just saying that to satisfy their bosses, who are under intense pressure to justify every dollar spent with at least one dollar returned.

IT executives working through this economic slump find themselves in a precarious position. CEOs, chief financial officers and shareholders have become hard-liners when it comes to technology ROI. In part, it's a backlash from the free-spending ways of the dot-com boom. Now they want proof that every IT investment pays back in some tangible way - no excuses.

But taking an accountant's view of IT priorities could actually be counterproductive, because a spreadsheet doesn't tell the whole story. In fact, some of the IT projects that impact business the most can't be measured easily, if at all, some experts now say.

"The boldest strokes I've seen in IT haven't had to go through these ROI tests," says Bob Reeder, CIO at Alaska Airlines Inc. He points to advancements such as General Motors Corp.'s OnStar road navigation system, which ushered in new levels of customer service and therefore doesn't fit neatly into traditional models for calculating ROI, he says.

If Seattle-based Alaska Airlines had ranked its IT priorities based purely on strict ROI metrics, it might not have started selling tickets online in 1995 or added check-in kiosks at airports in 1997 - two projects that have produced positive results, Reeder says.

Basing IT priorities on ROI rankings is a fool's game, Reeder says, largely because of the way so many ROI metrics are calculated. Under pressure to produce quantitative forecasts, project leaders have learned how to be creative with numbers.

"It's like the biggest liar wins," Reeder says.

While CIOs say the payback on most IT projects can be measured in dollars, many utilitarian but necessary efforts, like infrastructure upgrades and installing and supporting collaborative applications, can't be, and those projects are getting short shrift these days.

Indeed, ROI has a softer side that's too often ignored. This aspect of ROI helps explain why collaborative applications and IT projects that help improve product quality or customer service sometimes get the green light, despite their questionable short-term impact on the bottom line.

"There are soft benefits to every investment, in addition to the hard-dollar benefits," says John Berry, an independent management consultant in Bend, Ore., who specializes in assessing payback on software. Some projects produce qualitative results that are meaningful but difficult to trace back to the initial technology investment, he says.

"E-learning is supposed to create a smarter organization. How do you measure smarter?" Berry asks. And even if you could measure the collective intelligence of your staff, you couldn't necessarily attribute any intellectual gains to new technology.

It's particularly tricky to calculate the benefits of infrastructure improvements. Upgrades to servers, networking equipment and operating software contribute to the success of every application they touch, yet it's nearly impossible to know whether an application does more for the business if, say, it's aided by more bandwidth.

Those who try to apply ROI figures to infrastructure investments miss the point, says Victor Votsch, an analyst at V-Square Solutions, a Narberth, Pa.-based consultancy. "Do you [measure] ROI on your phone system?" he points out. "With infrastructure, you're preparing for change. You know change is a constant, but you're not sure what it will be."

Creative Calculations

Some IT shops, nonetheless, are able to find creative ways to quantify benefits that are largely qualitative. Cisco Systems Inc., for example, has been known to place a dollar value on customer satisfaction, based on the presumption that satisfaction leads to more sales.

But some analysts advise against this practice. If you can't trace a customer's satisfaction directly to a new sale, or a reduction in labor directly to a cut in head count, don't credit the technology for the ROI, says Rebecca Wettemann, an analyst at Nucleus Research Inc., a technology ROI consultancy in Wellesley, Mass.

IT shops are particularly creative when it comes to measuring the results of process improvements. If an application is deployed to automate expense reporting, a company might count the number of hours managers used to spend on their reports and multiply that number by the average hourly salary of those employees. That would likely produce a hefty ROI figure.

However, you don't necessarily get an hour of productivity back for every hour of labor you save, Wettemann says. In practice, managers and executive-level workers tend not to give all of their saved time back to the company. Instead, they may use the found time to check their stock portfolios online or make personal calls. So in this case, Wettemann might apply a "correction coefficient" to her ROI analysis that says the company will get only 30 minutes back for every hour of labor each executive saves, she says.

Reeder doesn't even try to calculate such gains. "A lot of people are kidding themselves that they can really quantify ROI on something as soft as productivity improvements that don't lead to lower head count," he insists. He calls such attempts to measure productivity gains "fun with numbers."

At GM, most IT projects produce calculable results, says Chief Technology Officer Tony Scott. For those that don't, particularly those projects that generate long-term but not short-term results, GM's IT department makes an extra effort to align with business managers in the regional or operational units affected by the project, Scott says.

The four regional CIOs at GM are closely aligned with their respective chief operating officers, in effect giving the CIOs responsibility over profit and loss (P&L) for their regions.

"You need a process owner and a P&L owner to say, 'I'm willing to take a bet on this,' " Scott says.


IT shops that aren't formally aligned with business management should always seek the support of a third party when arguing for a project that will produce soft benefits, Votsch says. These third parties can be independent analysts or consultants. "You need the credibility of a third party who doesn't have the perceived bias that may be there in the IT organization," Votsch says.

Regardless of how you lean on ROI measurements, it's clear that CIOs will continue to push hard for quantifiable metrics and returns.

"The motivation is out there to be as comprehensive as possible, to think of the totality of impact of technology purchases on a company," says Berry. "I believe the CIO when he says he attempts to measure everything. The question is whether he can."

Joachim is a freelance business and technology journalist in Port Jefferson, N.Y. Contact him at

Copyright © 2002 IDG Communications, Inc.

Shop Tech Products at Amazon