E-lusive Benchmarks

Computerworld - The all-too-visible fizzle of dozens of Internet companies in recent months has many executives demanding detailed status reports on just how their own e-commerce operations compare. But there's not a lot of data out there to measure against. At least not the kind of deep, historical benchmarking metrics that seasoned IT and corporate managers can gather to judge the performance of other functional areas, such as end-user training or software maintenance.

"There's a long history to compare and contrast yourself against in mainframe-oriented IT benchmarking, and even client/server," says Michael Lee, a systems consultant at AIT Global Inc., an IT education and seminar group in Kings Park, N.Y. "That's not the case with strategic Internet projects."

Senior IT managers typically don't share many stories about what has or hasn't worked for them because e-commerce is viewed as a potentially huge source of sales and profits. No one wants to give away their big-money secrets.

As a result, questions often go unanswered, such as, "How does our plan for tying a warehouse inventory system to an online ordering application stack up against the way others have done it?" or, "Do the three extra months my development staff takes to build new features for our shopping cart software give us any competitive advantage?"

Rather, the metrics most often used to measure the success of Internet projects are of a much smaller scale. For example, log files can be used to catalog who visits a site, when they visit and what they do there - data from which analysts can extrapolate whether a given link or set of pages is effective. The IT department can create survey boxes to solicit opinions from random Web visitors. Software packages can test how quickly pages load and whether links are broken.

Short on Answers

But these measures don't yield much concrete material about how a company's Internet efforts compare with those of its rivals and whether it spends IT money wisely.

A few companies, such as Keynote Systems Inc. in San Mateo, Calif., and Hackett Benchmarking and Research in Hudson, Ohio, have begun to sell consulting services to address these questions.

Hackett, for example, instructs user companies to collect specific data from business units and their IT groups about the performance of and costs associated with key applications. For instance, a recent Hackett study at TRW Inc., an automotive and aerospace products maker in Cleveland, required the company to collect data on such items as spending on hardware, software, networking and staff, as well as details