TPC prepares Web server benchmark revision

The Transaction Processing Performance Council is updating its specs

IT managers who consult the Transaction Processing Performance Council's benchmarks as a guide for purchasing decisions will have an updated metric later this year to use when selecting servers, according to representatives of the TPC.

Benchmarks remain an important tool in the corporate IT world for comparing system performance from vendor to vendor, and the TPC, one of the most influential benchmarking organizations in the industry, is updating its specifications to reflect current usage models.

The council plans to release a major revision to the TPC-W benchmark for Web servers, one- to eight-processor machines that generate Web pages and process transactions over the Internet, said Michael Majdalany, administrator of the TPC. TPC-W is currently undergoing a public review and is scheduled to be released by the end of the year.

Council members are also reviewing changes to the price/performance figures, which are hotly contested by vendors and scrutinized by customers. The TPC wants customers to be able to compare price/performance results across the major TPC benchmarks, something they are currently unable to do, Majdalany said.

The TPC has 21 full members, including Advanced Micro Devices Inc., BEA Systems Inc., Dell Inc., Hewlett-Packard Co., IBM, Intel Corp., Microsoft Corp., Oracle Corp., Sun Microsystems Inc. and Unisys Corp. About 40 organizations in total belong to the group, including a number of market research firms and user groups.

Four major benchmarks are currently administered by the TPC. TPC-C measures online transaction processing in large multiprocessor servers and is one of the most widely cited benchmarks in vendor marketing materials. In addition to TPC-W, the council sets guidelines for TPC-H and TPC-R. TPC-H provides decision support for environments that require ad hoc queries to a database, and TPC-R provides decision support for environments where the queries are known ahead of time.

Because of the potential marketing advantage of achieving superior benchmarks, the TPC requires that all results submitted be independently audited, said Michael Molloy, current chairman of the TPC and a senior manager at Dell.

Any company that wishes to publish a TPC benchmarking result must have that result audited to make sure that all of the hardware and software used is publicly available and not a "benchmark special," Molloy said. A detailed configuration of all the equipment used to produce the result must be published on the TPC's Web site along with the result.

No one is under the illusion that benchmarks are perfect, but having some metric with which to draw conclusions about various equipment is useful, said Gordon Haff, a senior analyst at Illuminata Inc. in Nashua, N.H. "Whatever the flaws of benchmarks, there is a value to having at least some relevant cross-industry comparisons," he said.

Those comparisons often help narrow a company's options when selecting a new vendor, said John Stevenson, vice president and CIO at Sharp Electronics Corp. in Mahwah, N.J.

"There may be moments where you use benchmarks to narrow down the field. You might be worried if [that vendor] is trailing, unless they are the incumbent," Stevenson said. A strong existing relationship with a vendor can offset less impressive benchmark results as long as the results meet a company's base performance requirements and the vendor has a history of good service and support.

Copyright © 2004 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon