What to Count?

There's no shortage of business and technical details to measure when determining technology's value to the business. The trick is picking the right things to quantify

"Simplify your life" is more than just the latest New Age mantra. It's advice well worth heeding when it comes to IT metrics. If a company is to have any shot at measuring IT's worth to the business, most experts agree, it would be smarter to rely on just a few well-chosen yardsticks than to drown in a sea of 500 data points.


The 1-2-3 Count

Rod Hall, vice president of consulting services at Chicago-based consultancy Compass America Inc., says there are three types of IT measurements, and a "smart CIO nails all three and doesn't confuse them." They are:

Metrics used to manage IT internally - network uptime, for example. These aren't very useful to upper management, but they do help a CIO track the costs of running the IT group and prove that it's an efficient operation.

Metrics for senior executives - a one-page report card, for example. If IT is considered a cost center, "you want to show how the costs compare to IT in similar companies," Hall says. The report card can also include IT services and costs broken out on a per-user basis, as well as a sense of the quality of those services and their value to users. Make sure to define quality and value in your report, Hall advises.

Metrics for business unit managers - detailed usage reports, for example. These managers also value forecast information so they can budget more accurately. "If I'm running 20,000 desktops in my business unit and consumption goes up by 10%, how will the costs change?" Hall says. "Those numbers need to be unique to each business unit."


But that's about all the experts agree on. By itself, measurement is no automatic guarantee that IT aligns with the business or gives value to it. For one thing, there's little consensus on which metrics should be used. ROI, that old financial stalwart, turns out to be squishy when it's used to measure IT's business value. Some other traditional financial measures don't work well either.

Revenue per employee, for example, doesn't take into account that a great deal of a company's IT operations or manufacturing and production work might be outsourced. As metrics expert and consultant Paul A. Strassmann points out, "If you look at the [profit and loss] statements of American corporations, 20% to 85% of the revenue is based on assembling components or integrating materials produced by humans in other companies." So, he says, one way to increase revenue per employee is "just don't make it, and move the people to someone else's P&L."

Welcome to the brave new world of IT metrics, where pretty much everything you thought you knew has changed. Sure, it's still important to measure things like network availability, compute cycles and even, if you must, lines of code. But those things alone aren't going to provide upper management, employees or stockholders with a better idea of what IT is delivering to the company's bottom line or overall success.

Instead, think big picture, most experts say. Find out what business managers and executives need to know based on corporate strategy and goals, and set up metrics and communications processes to tell them how IT is performing against those goals.

There are at least a half-dozen formal, recognized methods for tallying IT's worth, each with its own inherent strengths and weaknesses as well as consultants to back it. These range from the balanced-scorecard approach to the method of looking at each IT project as an investment. But the truth is that there's no simple way to measure the true value of IT that works for everyone.

Experts are also split on the true value of including intangible factors such as employee satisfaction as part of the equation in figuring out IT's worth. But even if

satisfaction can be measured, how does one attach a dollar value? Do happier employees produce more? For those reasons, some CIOs say that if something can't be measured, it shouldn't be part of any metric. There's no one answer that works every time for everybody.

Doug Turk, executive vice president at Inforte Corp., a consulting firm in Irvine, Calif., has conducted dozens of ROI analyses for customer relationship management (CRM) and e-marketplace systems, among others. He advocates measuring ROI for IT projects "just like measuring the value of any other investment you'll make." You must determine "what is the metric you're trying to improve - increase the number of sales leads or reduce the cost per lead, reduce the sales cycle time, improve the effectiveness of the sale and the size of the deal," he says. Once the goals are clear, dollar values can be assigned.

"Ask the hard business questions first," Turk advises. "What is the current sales cycle, and where are the opportunities to create business value? If the cycle time is 90 days and the average sale is $5 million, then you've got your baseline. This is where you're at, pre-technology-adoption."

Turk suggests that you always start with the business value you're after. "Implement the technology to support that, and then measure it," he says.

The idea is to identify a handful of core metrics that are meaningful to users and executives and then measure them from a set point in time. This is your baseline. Measure them the same way every three to six months, and report the results to all interested parties. Also, experts say, plan on spending 2% to 3% of your IT budget to do metrics right.

Steve Hightower, director of computing services at Lockheed Martin Corp. in Sunnyvale, Calif., says his company is doing just that. Lockheed's metrics prove, among other things, that the company saved about $30 million by revamping two IT call centers. "We're solving problems faster" and getting end users back to work more quickly, he says. For example, 70% of technical support calls are now resolved on the first try, up from 40%.

"We also don't have to have as many people in the field to solve problems, and we can apply those resources to more strategic IT projects," Hightower adds.

What the aerospace giant's metrics program is all about, he says, is trying to define the value of IT services in the same terms that customers do. This translates into money and time saved, for sure, but the company also tracks other things, such as additional ways IT projects can help employees get more done. "If a system is available 90% of the time for 500 people and we can get it to 95% availability, that's a measurable benefit," Hightower explains.

At Ingram Micro Inc. in Santa Ana, Calif., the focus is on measuring productivity, says Guy Abramo, chief strategy and information officer. "We're more into segmented accounting, activity-based costing - what employees do in their jobs, what keeps them busy," he says. The goal is to automate whatever will help employees do their jobs faster, better or easier. Also, he says, "the weighted cost of capital for this company is 12%, so we're looking only at IT projects that return more than that."

The weighted cost of capital, a standard accounting measure, takes into account interest rates, the company's tax rate, the risk inherent in the project or business, the shares outstanding and other factors. Each industry, and each company, has its own weighted cost of capital. The message: Any project or business that returns less than the weighted cost of capital isn't worth doing because it won't provide full investor value.

About two-thirds of all IT projects at Ingram Micro are justified on the basis of savings, with the last third driving new revenue. "We're really looking at ways to continue to drive selling, general and administrative expenses down," Abramo says. One project in the cost-cutting category, Ingram Micro hopes, is Orion, which helps the $30 billion electronics distributor's procurement staff manage the workflow of purchasing.

"We buy a lot of stuff, and the idea with Orion is to take the time and mystery out of how much to order," Abramo says. One important measure used here is the fill rate - the rate at which Ingram can fill a customer's order when it's placed. "I don't want to hold too much in inventory, but I want to fill the customer's order as quickly as possible," he explains. Orion's success isn't yet clear because it will take a year to get meaningful figures, he says.

In the category of revenue-generating IT projects, Ingram built a model to predict customer attrition, and it's correct 80% of the time. "Now I know who's vulnerable" and who might go to a rival, Abramo says. Ingram can now change its behavior before a customer decides to leave. But the company won't know the project's full impact until the model is put into production.

What about intangibles?

There are many ways to skin a metric. Some believe very strongly that only things that can be directly measured belong in any IT valuation. Others think that softer numbers, like customer satisfaction levels, are useful, too. Tom Madison, vice president of management at Inforte in Chicago, says the CEOs he talks with are very clear about wanting to know the rate of return of IT projects. "They say, 'Show me the bottom line, or stop talking,' " he says. "They want to know the impact on the operating margins" of the company, he adds.

For this reason, he says, companies should incorporate only tangible items into a financial analysis of a proposed IT project. "It's not impossible to quantify everything," Madison says, "but it does require some work. If you can't run the proposed results of your investment through the financials, it's probably an indication that something's wrong, that you haven't done enough analysis."

But Kosmo Kalliarekos, a partner at The Parthenon Group, a consulting firm in Boston, says intangibles can be a huge factor in "how to value IT externally, at the customer touch points. This is much harder to do - to understand what the customer really values in a service, offering or product. Do they value it enough not to switch from a competitor or to pay for it?" To the extent that an IT service forms the basis of a product or service or helps the company deliver it, those questions are relevant, he says.

But Kalliarekos emphasizes that even the intangibles have to be quantified to be of any use. "No matter what metric you're using, ask yourself how this ties to the business. If you can't tie it to the business, it's difficult to say if you're doing the right thing or the wrong thing. And that will result in investment in the wrong area or not enough investment," he says.

Experiences at Intel Corp. (more about Intel at www. computerworld.com/roi) and Siemens Business Services Inc. (SBS) in Rye Brook, N.Y., show this dichotomy pretty clearly. Intel's CIO says it's fine to take intangibles into account when measuring IT's contribution to the business. SBS's CIO disagrees.

Intel measures plenty of hard data, such as cost avoidance and compute cycles for its design engineers. But the company also includes intangibles, such as customer satisfaction and quality, in the equation of how well IT is doing in general.

"One of the things we've concluded, and this is something Craig Barrett, our CEO, has completely bought in to, is that a large percentage of the benefits from IT is not measurable in straight financial terms," says Doug Busch, Intel's CIO. "We're helping employees be more responsive to customers by giving them data they need to make better decisions and reduce decision-making time, being more accurate and digging deeper into things. These don't lend themselves to a bottom-line ROI number."

An example of how intangibles play into the mix is a company intranet Intel implemented in 1996. "Today, it seems like a no-brainer, but back then, it was leading-edge," Busch recalls. So the team that did the analysis to sell the intranet to senior management, a team he led, did the persuading with both tangible and intangible benefits. Among the former: a reduction in paper copies and the elimination of the paper company phone book and printing costs. "These weren't revolutionary, but they showed there was definitely a return," Busch recalls.

On the intangible side, there was "a whole level of benefits that were hard to quantify: sharing information in real time across the world vs. having to fax it or use FedEx; finding out information about our partners, customers and employees without having to make phone calls," Busch says. "This has revolutionized the way we do business."

But some things do translate into bottom-line benefits. Intel measures the WAN traffic and compute capacity for its design engineers, a major group of power users. "We know for any given compute and network capacity how many products our engineers can design," Busch says. The company also monitors the per-user cost-effectiveness of the workstations in that group. Intel switched the engineers' Unix workstations to Linux and has saved more than $100 million in the past four years.

Overall, the view at Intel is that "there's a high degree of management judgment in how we make our IT investments," Busch says.

At SBS, the feeling is different. The company, formerly known as Entex IT Service Inc., was acquired by Siemens AG in April.

"We're looking at very tangible things," says CIO Gerhard Cerny. SBS doesn't do ROI analysis for every IT component; for example, fundamental functions such as the data center are treated as costs of doing business. But for every new project, there's an ROI statement that includes the projected savings or revenue gains.

Once the ROI has been accepted by the business unit manager, that manager's budget is tweaked based on the benefit the technology will bring. For example, if a new CRM system is implemented with a promise to raise the sales force's productivity by 20%, the sales quota will then go up by 20%. "It's a twist on the ROI discussion," Cerny says, "because it means the business managers are bellying up to the bar. They accept a reduced budget or new business terms, and so they get involved in making sure that it does what we say it will do. It's a pretty good sanity check."

This system controls costs better and reduces projects' scope creep, Cerny says. Also, IT and business employees are motivated with bonuses and raises to keep on schedule and on budget. "All our livelihood, success and benefits depend on coming in on or ahead of this ROI projection. It's a painstaking prospect if you want to go over time and budget. Believe me, it's not something that most people want to engage in," he says.

"We're trying to build a culture," Cerny explains, "so that ROI is real and not just a number on paper."


Copyright © 2001 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon