VMware benchmark tool raises fairness questions

VMware Inc. today is making available a free benchmarking tool that IT professionals can use to evaluate the performance of virtualization technology in their data centers. Meanwhile, it is seeking to assure competitors that the results of tests using the VMmark tool will be fair to all of them.

VMmark is available for a free download. (Editor's note: as of posting time, around 11:35 a.m. on July 23, neither the download URL nor the URL with the press announcement about VMmark were working.)

The benchmark can be used to determine how well virtualization software works with any of six typical data center workloads: file server, e-mail server, Web server, a standby server, an online transaction processing (OLTP) database and a Java order entry system. VMmark runs on either Linux or Windows.

VMware shared a prototype of VMmark in October 2006 with members of Standard Performance Evaluation Corp. (SPEC), an industry group that establishes performance benchmarks so customers can evaluate products. At that time, SPEC had just formed a working group to develop benchmarks for virtualization software. It is still working on those benchmarks.

Though helpful for comparing various products, benchmarks are controversial because of concerns that the results can be manipulated to make one vendor's products look better than those of its competitors. SPEC and other industry benchmarking groups are organized to create fair benchmarks that will yield impartial results.

VMware acknowledges that there's skepticism about whether its benchmarking tool is a fair way to compare VMware virtualization technology with other brands, so it strove to be impartial, said Andrea Eubanks, senior director of enterprise and technical marketing at VMware.

VMware is represented on the 13-member SPEC working group that is developing the virtualization benchmark, along with competitors such as SWsoft Inc. and Trigence Corp. Major technology companies including Hewlett-Packard Co., Microsoft Corp. and Intel Corp. are also part of the group.

"We have created VMmark to be an open standard," said Eubanks. "In order to prevent misconceptions of what our intentions are, ... we presented our technology preview to SPEC so they could use it as the basis of the open-standards benchmark. We want to improve our product, we don't want to cook our benchmarks."

VMware is introducing VMmark while the SPEC panel is still developing its standards. "The process to becoming a standard is a quite long one," said Eubanks.

But VMware has not always been as impartial as it claims to be, said John Bara, vice president of marketing for XenSource Inc., a VMware competitor.

Bara says VMware released a white paper earlier this year comparing VMware's ESX Server hypervisor to XenSource's open-source Xen hypervisor on a Windows operating system, and the results showed VMware performed better. Bara says the comparison was not fair because open-source Xen is optimized to run best on Linux, not Windows. XenSource's Xen Enterprise product is optimized to run Windows and would be a more balanced comparison to VMware's ESX.

"We cooperate with VMware in a lot of areas, but we went back to them and said 'foul,'" said Bara. To VMware's credit, he said, it redid the test and accepted the revised benchmark that showed VMware ESX and Xen Enterprise with nearly comparable performances.

Also Monday, XenSource entered into an OEM agreement with Symantec Corp. to embed Symantec's Veritas Storage Foundation virtualized storage management software into XenSource's XenEnterprise virtualization hypervisor.

Copyright © 2007 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon