Linux vendors claim Forrester Report favored Microsoft

Say Forrester Research showed bias in study of responses to security flaws

Four major Linux distributors have sharply criticized a recent report in which Forrester Research Inc. found that Microsoft Corp. outperformed them in responding to and fixing security flaws.

In a joint letter released April 6, Linux distributors Debian, MandrakeSoft Inc., Red Hat Inc. and SUSE Linux AG questioned the validity of Forrester's conclusions and claimed that the report had "extremely limited real-world value" for users.

"It's bogus in its current form," said Joey Schulze, a member of Debian's security team.

Laura Koetzle, the author of the Forrester report, defended her company's analysis of the data. All vendors studied in the report were measured equally using publicly available vulnerability data and widely accepted vulnerability rating measures from the National Institute of Standards and Technology (NIST), she said.

Cambridge, Mass.-based Forrester's report "Is Linux More Secure than Windows?" was released on March 22. It looks at how Microsoft and the four Linux vendors responded to reports of security flaws from June 1, 2002, to May 31, 2003. Microsoft ranks first among the vendors for its "responsiveness" and its "thoroughness" in dealing with reported security vulnerabilities.

On average, Microsoft took 25 days between public disclosure and the release of a fix, and it was the only company to fix all vulnerabilities, the report stated. However, Microsoft also had the highest percentage of serious flaws.

In contrast, Moreno Valley, Calif.-based MandrakeSoft took 82 days on average to issue fixes for its Linux distribution, the Forrester report said. SUSE took 74 days, and Red Hat and Debian each took an average of 57 days.

The Linux vendors ranked lower than Microsoft in terms of the percentage of reported flaws they fixed. Red Hat, which fixed all but one flaw, was closest, while Debian ranked last, fixing 275 out of 286 flaws.

While the data that the analysis is based on is accurate, the conclusions are not, said Vincent Danen, security update manager at MandrakeSoft. By measuring only the time elapsed between public knowledge of a security flaw and the availability of a vendor's fix, the study failed to make a distinction between critical flaws and the not-so-severe ones, the jointly signed letter said.

Linux vendors typically treat flaws on a case-by-case basis, with high-risk flaws getting a higher priority than low-risk ones, Danen said. The response to a flaw is based on risk assessments made by each distributor and may not always coincide with the assessment made by a third party such as NIST, he said.

"Our users will know that for critical flaws, we can respond within hours," SUSE Linux said in a statement. SUSE is now owned by Novell Inc.

By focusing purely on quantitative analyses, the Forrester report fails "to differentiate between both the seriousness of the flaws and, more importantly, the quality of the fixes," SUSE said in its note.

Making a distinction between serious flaws and not-so-serious ones when devising a response is crucial, said Joe Poole, manager of technical support at Boscov's Department Stores LLC in Reading, Pa.

"[Vendors] have to separate the things they need to do tomorrow and what can be done in a month," he said.

This isn't the first time that a Forrester report comparing Microsoft products and Linux has been criticized. In September, the research firm drew criticism from users over a Microsoft-funded report that concluded that Microsoft development platforms had a substantial cost advantage over Linux/J2EE for portal applications (see story).

But Koetzle stressed that her security report had no Microsoft backing whatsoever. "Microsoft did not pay for this report," she said.

Copyright © 2004 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon