Microsoft bashes Windows XP, Vista benchmarking

Computerworld - Editor's note: The person quoted in this story as "Craig Barth" is actually Randall C. Kennedy, an InfoWorld contributor. Kennedy, who presented himself as the CTO of Devil Mountain Software, no longer works at InfoWorld. Given that he disguised his identity to Computerworld and a number of other publications, the credibility of Kennedy's statements is called into question. Rather than simply remove stories in which he is quoted, we have left them online so readers can weigh his data and conclusions for themselves.

Microsoft Corp. and a performance testing researcher are trading barbs over recent benchmarks that claim the unreleased upgrade for Windows XP runs Office faster than the upgrade slated to ship early next year for Vista.

According to tests run by Devil Mountain Software Inc., Windows XP Service Pack 3 (SP3) blows through the company's OfficeBench test suite about twice as fast as Vista SP1 does.

Nick White, a program manager for Microsoft's Vista team, bashed the benchmarking in a post to a Microsoft blog last week, and called Devil Mountain to task for testing before either operating system reached its "Release to Manufacturing" (RTM) stage, the so-called gold code that is actually duplicated on disc. "Publishing benchmarks of the performance of Windows Vista SP1 now wouldn't be a worthwhile exercise for our customers, as the code is still in development and, to the degree that benchmarking tests are involved, remains a moving target," White argued.

He also called into question OfficeBench, dismissing it as a "window-open, window-close routine at accelerated speed."

"It isn't representative of real-world user behavior and hence isn't an accurate gauge of the actual end-user experience," White said. "Tests like these only measure a very small set of Windows capabilities and so aren't representative of the user's overall day-to-day experience of working with Windows and running applications."

White's comments raised the hackles of Craig Barth, Devil Mountain's chief technology officer, who called out the Microsoft program manager in more than one place. "As the author of the OfficeBench test script I take personal offense at your cavalier handling of what is a very real and pressing issue for enterprise IT organizations," Barth said in a comment attached to White's blog. "Consider this fair warning that we will NOT let this issue rest and that you'll be hearing from us again in the near future."

"It was a hit-and-run job," Barth added today in a telephone interview. "We're formally challenging them to prove that the [OfficeBench] benchmark results aren't valid. If they can, I'd really like to see it."

Barth defended the test script as realistic and countered White's contention that benchmarks at the service pack stage are meaningless. "I think it was reasonable to test both release candidates at this point," he said. "My experience [with Windows] since 1991 is that if there's a performance difference of more than a percent or two between release candidate and RTM, someone at Microsoft [messed up].