Metrics vendor defends Windows 7 memory claims

'Everyone thinks they're a performance expert,' rebuts Devil Mountain CTO

Editor's note: The person quoted in this story as "Craig Barth" is actually Randall C. Kennedy, an InfoWorld contributor. Kennedy, who presented himself as the CTO of Devil Mountain Software, no longer works at InfoWorld. Given that he disguised his identity to Computerworld and a number of other publications, the credibility of Kennedy's statements is called into question. Rather than simply remove stories in which he is quoted, we have left them online so readers can weigh his data and conclusions for themselves.

The Florida firm that on Wednesday said most Windows 7 machines exhaust their physical memory, and as a result take a performance hit, defended its data and conclusions after naysayers dismissed its findings.

"Everyone thinks that they're a [Windows] performance expert," said Craig Barth, the chief technology officer of Devil Mountain Software, a performance metrics software maker. "They look at their PC and say, 'My PC doesn't do that.'"

Barth was reacting to the firestorm of criticism over his claim that 86% of the Windows 7 PCs among the 23,000 tracked by Blue Mountain's community-based Exo.performance.network (XPnet) exhibit signs of severe and sustained memory exhaustion.

Readers of Computerworld's story of Wednesday repeatedly said, sometimes stridently, that they thought XPnet's data was bunk. "Vague reports that a little-known outfit somehow has managed to get metrics on 23K machines, and has found their memory utilization to be abnormally high without explanation as to how, is not a reliable source," charged Micah Haber in an e-mail to Computerworld.

More than one reader accused XPnet of spreading FUD (fear, uncertainty and doubt). "A good operating system will not have much free RAM at all, but instead will allocate the unused RAM to buffers and caching," argued Kevin Pieckeil Thursday. "Having only a few megabytes of 'free' RAM doesn't mean the system's memory resources are being exhausted."

Countless readers, including Haber, did just what Barth pointed out: They cited numbers from their machines. "Anecdotally, both of my machines, one having previously run XP, the other Vista, exhibit lower memory utilization under Windows 7," said Haber. "Both have 2GB of RAM, and usually are using between 38% to 52% of the total amount of physical RAM available."

Others were more blunt. "There is so much wrong with this article I don't even know where to start," said someone identified as "cregan89" in a kick-off comment on Digg . "I simply do not believe for a second the figure they give that '86% of Windows 7 computers use 90-95% of available RAM'," continued Cregan89. "So I'm going to go as far as to call this article a downright lie."

Barth defended the data and his conclusions, arguing that Blue Mountain was hardly a "little-known outfit" as he ticked off his and his company's bona fides. The benchmark software, dubbed DMS Clarity Suite, was created by former Intel performance engineers, and is deployed commercially at financial firms, he said, including Morgan Stanley and Credit Suisse First Boston, as well as on Wall Street trading floors, where PC performance problems may mean millions down the drain.

The software agent voluntarily installed by the members of XPnet is a stripped-down version of DMS Clarity Suite that's been tweaked for an average user's PC.

In a long entry on the XPnet blog Thursday, Barth dove into the details of the methodology of Blue Mountain's "Windows Composite Performance Index" (WCPI), a combination metric that includes a measurement called "peak memory pressure," which was what Barth cited in the commentary on Windows 7.

"Our methodology is straightforward," said Barth. "We monitor critical Windows performance counters, evaluate them against a series of weighted thresholds, and flag those events wherein one or more of the counters exceeds its configured target limit."

The peak memory value is calculated by comparing four different Windows performance values, including the Memory\Committed Bytes counter, the Memory\Pages Input per second counter, the PageFile\% Usage counter and an event duration value that tracks the length of time each value remained at or above a threshold set by Devil Mountain. The individual ratios are then weighted and combined to create the single number, said Barth.

It's much more complex, and ultimately more accurate, he argued than the Task Manger, the simplest Windows performance tool called up by most users to evaluate memory consumption. Barth said Devil Mountain's data was more analogous to that provided by Perfmon.exe, a more advanced measurement utility included in Windows.

Peak memory pressure is a "gradual metric," said Barth, explaining that it was not a snapshot -- as a look at Task Manager or even Perfmon would be -- but is a continuous series of data points that's "indicative of how much you're saturating the RAM." The XPnet agent records a data point every second, then averages each 60 seconds' values, resulting in more than 10,000 reported memory values each week.

"That eliminates the peaks and valleys," said Barth. "This is a slow-moving metric that doesn't jump up and down, but grows gradually over time as the PC is running."

And it accurately describes the reality of PC use and performance, Barth maintained. As a Windows computer runs throughout the day and consumes most, if not all, of its available physical memory, pressure builds on the Windows virtual memory manager to reshuffle the physical memory deck and page portions of certain processes to the hard disk. And swapping to the hard drive is a performance hit.

"Virtual memory activity slows up machines," Barth said. "Wall Street traders can't afford that, not when a delay of a few seconds could put millions on the line," he added, referring to Devil Mountain's commercial clients.

A PC with a high peak memory pressure value would feel sluggish to the user, Barth promised. "Oh, you'll feel it," he said. "This is going to vary PC by PC, sure, but clearly, for the thousands of machines we're monitoring, Windows 7 has a high level of memory saturation, or use, with significant durations. And memory saturated Windows 7 PCs happen a lot more and with greater severity than XP," Barth said, repeating his earlier contention.

Barth initially reported that XPnet's data showed that 86% of Windows 7 PCs met or exceeded the peak memory pressure, or memory exhaustion, threshold, compared to just 40% of the Windows XP machines the network monitors.

"Outside of Microsoft, I don't think anyone knows more about Windows performance than us," said Barth, reacting to readers like Cregan89 who dismissed the memory data as bogus. "No other industry metric provides equivalent insight into the runtime behavior of Windows systems as they exist in the real-world. If I was an IT manager, and managing Windows 7 PCs, or Vista for that matter, I would be concerned enough to look into [memory exhaustion]."

Microsoft, which collects an astounding amount of data from Windows machines -- the company used its telemetry capabilities with testers during Windows 7's development, for instance -- declined to comment on Devil Mountain's data or Barth's conclusions.

Users who want to compare their computers to the current WCPI numbers can do so by registering with XPnet and then installing the DMS Clarity Tracker Agent from Devil Mountain's site.

Gregg Keizer covers Microsoft, security issues, Apple, Web browsers and general technology breaking news for Computerworld. Follow Gregg on Twitter at @gkeizer or subscribe to Gregg's RSS feed . His e-mail address is gkeizer@ix.netcom.com.

FREE Computerworld Insider Guide: IT Certification Study Tips
Join the discussion
Be the first to comment on this article. Our Commenting Policies