Sidebar: CIOs Hold Market Research Firms to a High Standard

October 13, 2003 (Computerworld)

A whopping 87% of 133 respondents to a Computerworld survey said they have questioned the statistical validity or integrity of market research. And the same percentage said they would like to see clearly stated ethics policies regarding the research firms' vendor relationships.

"Like any other companies, I understand that they have to go after different sources of funding," said Cathy Brune, senior vice president and chief technology officer at Allstate Insurance Co. in Northbrook, Ill. "But they need to be honest about saying who pays for the research."

"If the research is funded by a vendor and it examines their marketplace, I question it with a jaundiced view," said Lew Temares, vice president of information resources at the University of Miami in Coral Gables, Fla.

Bruce Fadem, vice president and CIO at pharmaceutical company Wyeth in Paoli, Pa., said his biggest concern with market researchers isn't the credibility of their work but the kind of influence they can sometimes wield in establishing themselves as market makers.

Four years ago, Fadem made extensive use of research from one firm that he declined to name to help him make a purchasing decision on a global change management system. Fadem said the research firm in question "became infatuated" with a particular vendor in this market "and didn't do as thorough a job as they should have" in examining the viability of the company. "Nor did we," he acknowledged.

Cathy Brune, senior vice president and CTO at Allstate Insurance Co.

Two years after Wyeth had made a "major" investment in software from this company, the vendor—which was not a start-up—went out of business, said Fadem.

"There's a real risk with these information suppliers, because they become of significant size and they're in a position to make a market," said Fadem, who continues to subscribe to reports from Gartner Inc., Meta Group Inc. and Giga Information Group Inc. Market researchers "can tout a particular product and have their subscribers go in that direction at the cost of some very good products and services that don't get the support they deserve," he said.

Robert Schwartz, CIO at Matsushita Electric Corporation of America's Panasonic Co. division in Secaucus, N.J., said he has been a Gartner subscriber for years. And while he continues to rely on the firm's research to help him decide on IT product purchases, Gartner's expansion into consulting and other areas "was getting beyond research, and we found that we wanted to work with a firm that provided pure research," he said. Panasonic recently became a client of AMR Research Inc., "since it was purely focused on research," Schwartz said. The company also wanted to tap AMR's supply chain expertise, he added.

	Joe Puglisi, CIO at Emcor Group Inc.

Several CIOs touted the integrity of Gartner's research. "I'm pleased with the general research results we've gotten from Gartner, both the written results and from their analysts in particular. They've done a good job for us over the years," said Richard Gius, senior vice president of IT and CIO at Cardinal Health Inc.'s Medical Products and Services group in McGaw Park, Ill.

Curtis Wolfe, CIO and director of the IT department for the state of North Dakota in Bismarck, said the state has contracts with both Gartner and Meta Group. He said he finds the two firms "useful for affirmation" after his IT team evaluates market trends and product strategies by studying trade journals and attending industry conferences.

For instance, Wolfe said he recently asked Meta Group analysts for their insights on J2EE vs. .Net. "We found that their technical evaluations of the market were productive," said Wolfe. "They weren't necessarily supporting one over the other, and we found them to be very objective."

Joe Puglisi, CIO at Norwalk, Conn.-based Emcor Group Inc., made similar comments about Giga Information Group. "I think their opinions are based on fact and not on sponsorship," said Puglisi. "They're calling the shots as they see them."