Is it OK for Microsoft to bar benchmark results?

To summarize the tale briefly, independent lab Competitive Systems Analysis (CSA) of Danville, Calif., recently ran a series of tests based on SQL Server to compare the performance of Windows 2000 vs. Windows NT. We don't know the results except that CSA Research Director Randall Kennedy was surprised enough by them that he thought he must be doing something wrong. Microsoft engineers worked with him for five days to find the problem without success. Then, knowing that Network World was interested in including Randall's results in a story, Microsoft officials trotted out the no-benchmarks term from the SQL Server click-wrap agreement. Kennedy briefly considered defying them but decided not to get in a legal fight with Microsoft over tests that no one was paying him to do.

Microsoft's Murchie says the only purpose of the benchmarks restriction is to ensure that the public gets accurate information. It's standard practice in the database industry, he says, where vendors have worked together to develop standardized tests with repeatable results, and Microsoft is not the first database company to block publication of benchmarks.The license language is intended to keep labs and publications from rushing to print what might be misleading information and to prevent scurrilous competitors from using finagled test results in their advertising.

Let me say that it is not my intention to demonize Microsoft on this issue, because this is not just about them. In fact, I was delighted to see Murchie's letter because at least it meant a vendor was going on record about this policy. You may remember that a few weeks ago Network Associates refused to comment on the rationale behind a term in its VirusScan click-wrap agreement prohibiting publication of benchmarks and reviews without company permission. And Murchie is right that Microsoft isn't the first database vendor to try to enforce this restriction. Oracle has been threatening customers and publications with its benchmark restriction for more than a decade and in some cases has succeeded in squelching reviews.

But Murchie says that Microsoft's benchmark restriction is not meant to prohibit customers from sharing benchmark information with each other, which they can do without getting Microsoft's permission. As written, the SQL Server license itself would seem to apply to customers as much as to publications, but Murchie told me he plans to advocate changing the language to make it clear that it applies only to publishing benchmarks.

All that said, what Microsoft has done in this case is a very big step down an extremely slippery slope. Perhaps Kennedy's results were inaccurate, but Microsoft never proved they were. And who gave database vendors the right to judge whether a benchmark of their product is accurate?

That's what really worries me about Microsoft's actions here. After all, Kennedy wasn't benchmarking the performance of two databases; he was just using a database as part of a test comparing two versions of Windows. Think about the implications of that. The Windows licenses don't have the benchmark restriction, so will Microsoft now decide it needs that language in all of its end-user license agreements? And will that mean that anyone wanting to test the performance of a product that uses Microsoft software be required to get Microsoft's blessing first?

Taken to its logical conclusion, Microsoft's claim that it has the right to keep Kennedy from disclosing his results is tantamount to saying the company can keep Road&Track from testing the performance of a car that has a little bit of Microsoft software embedded somewhere.