Chinese antivirus firm Qihoo 360 Technology today was censured by three major testing organizations for cheating on the evaluations.
The three antivirus testing labs -- AV‐Comparatives, AV‐TEST and Virus Bulletin, of Austria, Germany and the U.K., respectively -- stripped Qihoo of all awarded certifications and rankings for this year. They will also put in place controls to make sure that Qihoo or others cannot "game" the tests in the future.
Qihoo denied the charges, saying they were "without merit."
Qihoo 360 is headquartered in Beijing, and reported revenue of $1.4 billion in 2014. Although the company's security products are little used outside of China, inside the People's Republic (PRC) the company claimed almost 750 million people used its free mobile security app, 360 Mobile Safe, last year. Qihoo is also known for its 360 Browser, which relies on Microsoft's Trident rendering engine, the same that powers Internet Explorer (IE).
AV‐Comparatives, AV‐TEST and Virus Bulletin concluded that Qihoo had provided a customized version of its security software to them -- they received the software from the Chinese company because the programs are not widely available outside the PRC -- that replaced the company's own antivirus engine with the one created by Romanian security firm BitDefender.
"After several requests for specific information on the use of third‐party engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users," the three labs said in a joint statement (download PDF).
That skewed the results in Qihoo's favor. "According to all test data, [Quihoo using its own engine] would provide a considerably lower level of protection and a higher likelihood of false positives," the labs said.
Prior to acknowledging the AV engine swap, Qihoo pointed fingers at two of its rivals, Baidu and Tencent, both also based in the PRC. Although Qihoo's allegations proved accurate, the testing organizations concluded that it could find no evidence that their actions -- setting code flags marked with the names of several test labs, which in turn implied "some difference in product behavior" -- gave them a significant advantage. "Both firms were able to provide good reasons for including these flags in their products," the labs added.
Qihoo's security products include similar flags.
"Users rely on independent results to make an educated decision regarding their protection software," argued Maik Morgenstern, AV‐TEST's CEO, in the statement. "If vendors start to manipulate the testing process, they are hurting everyone involved."
Qihoo has been criticized before for manipulating numbers. While it claims that the 360 Safe Browser has a majority share of China's market, analytics firms like Ireland's StatCounter say different: In March, Qihoo's browser accounted for just half a percentage point of user share, a measurement of how active each browser's users are on the Internet.
Critics have attributed Qihoo's claim of share dominance to shady practices, including making it difficult to uninstall the browser, overtly trying to convince users not to make an alternate browser the device's default, and evidence that removing 360 Safe Browser often cripples Internet connectivity.
Today, Qihoo took to Facebook to rebut the cheating charges, although it did not directly address the labs' claim that Qihoo switched AV engines.
"We believe the accusation and subsequent action ... is without merit," Qihoo said, then launched into a long discussion of why the Chinese security market was different from those in the West, making the tests inherently unfair.
"For example, many popular software add-ons in China that are flagged as malware by [the labs] are in fact performing proper functions and not malicious," Qihoo said. "A security product that strictly follows [the labs'] testing environment rule[s] could be rendered useless in China due to the significantly different real-world environment.
"As a result of our efforts, China has become the safest Internet environment in terms of the malware infection ratio, according to a Microsoft study," Qihoo concluded. "We certainly intend to continue to do so with or without lab testing scores."