Opinion: Can we take the Storage Performance Council seriously?

Computerworld - Recently, the Storage Performance Council (SPC) posted two benchmarks to its Web site that were run on an EMC Clariion CX3 Model 40. What's a bit unusual in this case is the fact that EMC did not run the benchmark. As many of you may know, EMC is not an SPC member, nor does it endorse the SPC benchmark. Rather, Network Appliance ran the benchmark on the Clariion and submitted the results. NetApp also ran the same benchmark against its own fabric-attached storage (FAS) devices, and followed up with a press release proclaiming (surprise, surprise) that a NetApp FAS3040 had just beat the pants off a Clariion.

Oddly enough, EMC may have brought this on itself. Noted EMC blogger Chuck Hollis (writing in a blog post titled "Does Anyone Take the SPC Seriously?") challenged anyone to run an SPC benchmark against Clariion by saying "We've never done an SPC test, and probably will never do one. Anyone is free, however, to download the SPC code, lash it up to their Clariion, and have at it." Reminds me of the old adage: Be careful of what you wish for.

We all know that, between competing storage vendors, all seems fair in love and war. So I'm not going to get between these two behemoths. Who's got the hotter box is not the point of this little blurb.

Rather, I'm curious about the behavior of the SPC, which changed its publicly stated policy with regard to benchmark results submitted on another vendor's product while the benchmark proceeded forward. Under the new policy, the submitting vendor must produce a full disclosure report. Once that report has been accepted by the SPC, the results are published to the SPC's Web site and become fodder for its member's press releases. As of January 29, EMC had 60 days to respond to the report.

My issue with the SPC, whose members are storage vendors, is whether or not we can regard it as unbiased and independent. In spite of the fact that we have here the unprecedented situation where vendor X does a benchmark run on a competing vendor's product, the SPC made no public statement with regard to the operative policy under which this benchmark was published. Rather, its administrator participated in the aforementioned NetApp press release that was published on the same day he claims he informed EMC of the benchmark results. Behavior like this makes Chuck's snarky blog item all the more relevant.

John Webster is the principal IT adviser at research firm Illuminata Inc. He is also the author of numerous articles and white papers on a wide range of topics and is the co-author of the book Inescapable Data: Harnessing the Power of Convergence (IBM Press, 2005). Webster can be reached at jwebster@illuminata.com.

Read more about storage in Computerworld's Storage Knowledge Center.