Recently published reports that have prompted questions about the credibility of some market research firms have spurred two of the biggest names in the field to make substantial changes to their ethics and disclosure policies.
And that's a good thing, users say. They want to see market research firms state their ethics policies more clearly on their Web sites and through other distribution channels, according to an exclusive Computerworld survey of 133 IT professionals that was conducted last week.
Last month, some IT professionals reacted angrily to a Microsoft-funded report released by Cambridge, Mass.-based Forrester Research Inc. that concluded that developing and deploying Web-based portal applications is substantially less expensive using Microsoft technology than it is using a Linux/J2EE combination .
Two weeks later, the release of an anti-Microsoft security report led to the firing of one of its co-authors, who harshly criticized the vendor's dominance of the software industry .
 |
 |
| Russ Lewis, CIO at GFInet Inc. | |
Interviews with 15 CIOs last week indicated that many IT executives continue to rely on market research firms as one component of making product purchasing decisions or setting strategic plans. But some CIOs say they have soured on the use of market research because of credibility concerns, high costs and doubts about the reports' value.
For instance, Russ Lewis said he made use of research from Forrester and Gartner Inc. quite often when he was CIO at New York investment bank Jefferies & Co. from 1994 to 1999. But he "found their research to be somewhat limiting and narrow in focus," said Lewis, who is now CIO at GFInet Inc., a subsidiary of New York-based brokerage services provider GFI Group Inc.
"I'm not a big fan of any of those market research firms," Lewis said. "I don't believe them to be independent, and I don't believe their research to be valuable except at a very high level for very large firms."
For its part, Forrester is making substantial changes to its ethics and disclosure policies following the release of the Microsoft-sponsored report and controversy surrounding another recent report that was funded by PeopleSoft Inc.
PeopleSoft had sponsored a survey of more than 600 business and IT users that examined users' satisfaction with products from several enterprise application vendors, including PeopleSoft, SAP AG, Oracle Corp. and Siebel Systems Inc. PeopleSoft put out a press release on Sept. 15 in which it boasted that it "outscored SAP, Oracle, and Siebel in the overall ability to deliver a superior enterprise application ownership experience" . The release attributed the findings to "an independent research study completed by Forrester Research" but failed to mention that PeopleSoft had paid for the study.
|
 |
|
George Colony, CEO at Forrester Research Inc. |
Steve Swasey, director of corporate public relations at PeopleSoft, said there was no point in disclosing the source of the funding. "What's the need? The research was objective and unbiased," he said. "We would have done the same if the research was sponsored by someone else."
Allowing PeopleSoft to publish the results in the press release was a mistake, a Forrester spokeswoman said last week.
In an interview last week, Forrester CEO George Colony said that the company has taken steps to "tighten" its internal processes and its integrity policy. In fact, Forrester "will no longer accept projects that involve paid-for, publicized product comparisons," according to a statement from Colony that's now posted on the company's Web site.
"We erred in the Microsoft and PeopleSoft cases, and we're correcting our processes," Colony said.
Forrester isn't alone. Boston-based Aberdeen Group Inc. plans over the next 45 days to "sharpen and enhance" the ethics policies posted on its Web site in order to more clearly state its research methodologies, said James L. Bedard, who became Aberdeen's president and CEO in August. The decision is partly a response to a June 2002 Wall Street Journal story criticizing firms that offer "praise for pay." In that story, the previous Aberdeen management defended the practice of publishing favorable research reports for vendors.
"It's critical that [customers] have the highest degree of belief in our integrity," said Bedard.
Executives at other leading market research firms, including Gartner, Meta Group Inc. and IDC, say they haven't made any recent changes to their ethics policies or how they go about disclosing them.
"If anything, we're often asked [by vendors] to change [our disclosure policies] in the other direction" and make them less restrictive, said David Yockelson, executive vice president and director of technology research services at Stamford, Conn.-based Meta Group.
Meta Group "rarely" takes on research sponsored by a single vendor, said Yockelson. And though the vendor does have the right to review Meta Group's findings in these types of reports, Yockelson said, "that doesn't give the vendor the right to not publish it or change it. They can come back and say they disagree. But we won't withhold it because it's funded by the vendor."
If Meta Group is conducting internally funded research in which a particular vendor receives a "substantial mention," the researcher will allow the vendor to fact-check the material and recommend changes for the sake of accuracy.
"We don't offer a vendor the guarantee that if they don't like what we write they can change it. As long as we are factually correct, we will publish whatever it is that we've created," said Yockelson.
Jim Shepherd, senior vice president at AMR Research Inc. in Boston, said vendor-sponsored research isn't an issue for his firm, at least from an ethics standpoint. "If we do research for a [vendor] client, it is not published and can only be used by a client internally," he said. "And we don't do much of that."
Like Meta Group, AMR will also let vendors fact-check its so-called deep-dive product review reports. "But we won't cede editorial control in any sense," said Shepherd.
Framingham, Mass.-based IDC, a sister company of Computerworld, "always discloses" in writing on the research document who the source of funding is for a particular piece of research, said CEO Kirk Campbell.
In the rare instances IDC does conduct vendor-sponsored research, said Campbell, the vendor has the right to review the findings. "But IDC has the final determination on all of its research content and how and where it is published," he said, adding that IDC hasn't made any changes to its research objectivity policies since they were established in the early 1990s.
Joseph Baylock, group vice president of vendor relations at Stamford, Conn.-based Gartner, said the company doesn't conduct any vendor-sponsored research, although its Gartner Consulting unit does conduct proprietary studies for vendors and groups of vendors. But those reports aren't published for public consumption, he said.
 |
||||||||||||||||||||||
|
|
|
||||||
Source: 133 respondents to Computerworld survey of senior IT managers |
