Microsoft bashes Windows XP, Vista benchmarking
Tester blasts back, challenges Microsoft to put up or shut up
Computerworld - Editor's note: The person quoted in this story as "Craig Barth" is actually Randall C. Kennedy, an InfoWorld contributor. Kennedy, who presented himself as the CTO of Devil Mountain Software, no longer works at InfoWorld. Given that he disguised his identity to Computerworld and a number of other publications, the credibility of Kennedy's statements is called into question. Rather than simply remove stories in which he is quoted, we have left them online so readers can weigh his data and conclusions for themselves.
Microsoft Corp. and a performance testing researcher are trading barbs over recent benchmarks that claim the unreleased upgrade for Windows XP runs Office faster than the upgrade slated to ship early next year for Vista.
According to tests run by Devil Mountain Software Inc., Windows XP Service Pack 3 (SP3) blows through the company's OfficeBench test suite about twice as fast as Vista SP1 does.
Nick White, a program manager for Microsoft's Vista team, bashed the benchmarking in a post to a Microsoft blog last week, and called Devil Mountain to task for testing before either operating system reached its "Release to Manufacturing" (RTM) stage, the so-called gold code that is actually duplicated on disc. "Publishing benchmarks of the performance of Windows Vista SP1 now wouldn't be a worthwhile exercise for our customers, as the code is still in development and, to the degree that benchmarking tests are involved, remains a moving target," White argued.
He also called into question OfficeBench, dismissing it as a "window-open, window-close routine at accelerated speed."
"It isn't representative of real-world user behavior and hence isn't an accurate gauge of the actual end-user experience," White said. "Tests like these only measure a very small set of Windows capabilities and so aren't representative of the user's overall day-to-day experience of working with Windows and running applications."
White's comments raised the hackles of Craig Barth, Devil Mountain's chief technology officer, who called out the Microsoft program manager in more than one place. "As the author of the OfficeBench test script I take personal offense at your cavalier handling of what is a very real and pressing issue for enterprise IT organizations," Barth said in a comment attached to White's blog. "Consider this fair warning that we will NOT let this issue rest and that you'll be hearing from us again in the near future."
"It was a hit-and-run job," Barth added today in a telephone interview. "We're formally challenging them to prove that the [OfficeBench] benchmark results aren't valid. If they can, I'd really like to see it."
Barth defended the test script as realistic and countered White's contention that benchmarks at the service pack stage are meaningless. "I think it was reasonable to test both release candidates at this point," he said. "My experience [with Windows] since 1991 is that if there's a performance difference of more than a percent or two between release candidate and RTM, someone at Microsoft [messed up].
- 15 Non-Certified IT Skills Growing in Demand
- How 19 Tech Titans Target Healthcare
- Twitter Suffering From Growing Pains (and Facebook Comparisons)
- Agile Comes to Data Integration
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- Taking Windows Mobile on Any Device Taking Windows applications mobile has many advantages, but the process of identifying a solution is complex. Learn how to solve this complex problem...
- Pay-as-you-Grow Data Protection: IBM Tivoli's Full-featured Data Protection Suite for Small to Medium Businesses IBM Tivoli Storage Manager Suite for Unified Recovery gives small and medium businesses the opportunity to start out with only the individual solutions...
- Streamline Data Protection with IBM Tivoli Storage Manager Operations Center IBM Tivoli Storage Manager (TSM) has been an industry-standard data protection solution for two decades. But, where most competitors focus exclusively on Backup...
- Simplify and Consolidate Data Protection for Better Business Results Learn about IBM® Tivoli® Storage Manager Operations Center, which provides advanced visualization, built-in analytics and integrated workflow automation features that leapfrog traditional backup...
- Meg Whitman presents Unlocking IT with Big Data During this Web Event you will hear Meg Whitman, President and CEO, HP discuss HAVEn - the #1 Big Data platform, as well...
- The New Way to Work Knowledge Vault This Knowledge Vault focuses on how, in today's increasingly virtual world, it's more important than ever to engage deeply with employees, suppliers, partners,... All Windows White Papers | Webcasts