Antivirus test labs call out Chinese security company as cheat

Chinese antivirus firm Qihoo 360 Technology today was censured by three major testing organizations for cheating on the evaluations.

Chinese antivirus firm, Qihoo 360 Technology, has been censured by three major testing organisations for cheating on the evaluations.

The three antivirus testing labs - AVComparatives, AVTEST and Virus Bulletin, of Austria, Germany and the UK, respectively - stripped Qihoo of all awarded certifications and rankings for this year. They will also put in place controls to make sure that Qihoo or others cannot "game" the tests in the future.

Qihoo denied the charges, saying they were "without merit."

Qihoo 360 is headquartered in Beijing, and reported revenue of $US1.4 billion in 2014. Although the company's security products are little used outside of China, inside the People's Republic (PRC) the company claimed almost 750 million people used its free mobile security app, 360 Mobile Safe, last year. Qihoo is also known for its 360 Browser, which relies on Microsoft's Trident rendering engine, the same that powers Internet Explorer (IE).

AVComparatives, AVTEST and Virus Bulletin concluded that Qihoo had provided a customised version of its security software to them -- they received the software from the Chinese company because the programs are not widely available outside the PRC -- that replaced the company's own antivirus engine with the one created by Romanian security firm BitDefender.

"After several requests for specific information on the use of thirdparty engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users," the three labs said in a joint statement (download PDF).

That skewed the results in Qihoo's favor. "According to all test data, this would provide a considerably lower level of protection and a higher likelihood of false positives," the labs said.

Prior to acknowledging the AV engine swap, Qihoo pointed fingers at two of its rivals, Baidu and Tencent, both also based in the PRC. Although Qihoo's allegations proved accurate, the testing organizations concluded that it could find no evidence that their actions -- setting code flags marked with the names of several test labs, which in turn implied "some difference in product behavior" -- gave them a significant advantage. "Both firms were able to provide good reasons for including these flags in their products," the labs added.

Qihoo's security products include similar flags.

"Users rely on independent results to make an educated decision regarding their protection software," argued Maik Morgenstern, AVTEST's CEO, in the statement. "If vendors start to manipulate the testing process, they are hurting everyone involved."

Qihoo has been criticized before for manipulating numbers. While it claims that the 360 Safe Browser has a majority share of China's market, analytics firms like Ireland's StatCounter say different: In March, Qihoo's browser accounted for just half a percentage point of user share, a measurement of how active each browser's users are on the Internet.

Critics have attributed Qihoo's claim of share dominance to shady practices, including making it difficult to uninstall the browser, overtly trying to convince users not to make an alternate browser the device's default, and evidence that removing 360 Safe Browser often cripples Internet connectivity.

Today, Qihoo took to Facebook to rebut the cheating charges, although it did not directly address the labs' claim that Qihoo switched AV engines.

"We believe the accusation and subsequent action ... is without merit," Qihoo said, then launched into a long discussion of why the Chinese security market was different from those in the West, making the tests inherently unfair.

"For example, many popular software add-ons in China that are flagged as malware by [the labs] are in fact performing proper functions and not malicious," Qihoo said. "A security product that strictly follows [the labs'] testing environment rule[s] could be rendered useless in China due to the significantly different real-world environment.

"As a result of our efforts, China has become the safest Internet environment in terms of the malware infection ratio, according to a Microsoft study," Qihoo concluded. "We certainly intend to continue to do so with or without lab testing scores."

Join the CSO newsletter!

Error: Please check your email address.

Tags Qihoo 360Malware & VulnerabilitiesantispamBullsecurityMicrosoftbitdefender

More about AVCBitDefenderFacebookIreland'sMicrosoftPRCTechnologyTridentWest

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Gregg Keizer

Latest Videos

  • 150x50

    CSO Webinar: Will your data protection strategy be enough when disaster strikes?

    Speakers: - Paul O’Connor, Engagement leader - Performance Audit Group, Victorian Auditor-General’s Office (VAGO) - Nigel Phair, Managing Director, Centre for Internet Safety - Joshua Stenhouse, Technical Evangelist, Zerto - Anthony Caruana, CSO MC & Moderator

    Play Video

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

More videos

Blog Posts

Market Place