Tencent, Qihoo antimalware firms are accused of cheating, stripped of rankings in antivirus tests

Several independent testing firms claim the accused companies submitted programs specially tuned to ace the benchmarks.

Chinese vendors Tencent and Qihoo have been accused of gaming antimalware tests by releasing optimized versions of their products for testing purposes that wouldn't be reflected in the real world, several antimalware firms said.

Both companies will be stripped of their rankings, AV-Comparatives, AV-TEST and Virus Bulletin said. The three stripped the rankings of Qihoo last week, and of Tencent on Monday night.

Both Tencent and Qihoo finished among the top six antimalware vendors in AV-Test results released in March, which ranked the firms on protection, the performance of the engines, and how usable they were. But they will apparently no longer appear on the official list, just as a college basketball team can be retroactively stripped of their victories and individual awards if improprieties are involved.

Why this matters: No one likes a cheater, but cheating on antivirus benchmarks potentially puts users at risk, if the product they choose doesn't actually provide the protection it seems to. The testing firms are also protecting their own credibility in rating these products.

'Inappropriate behavior'

"Today, three of the world's most renowned and trusted security testing bodies, AV-Comparatives, AV-TEST and Virus Bulletin, stand united to censure a security vendor after finding the firm submitted products for comparative and certification testing which behaved significantly differently from those made available to its users and customers," the three firms said in a joint statement posted to AV-Comparatives' Facebook page, in censuring Qihoo.

The three companies published a joint statement on the "inappropriate behavior" of Qihoo 360. In the tests, the antimalware organizations claimed, all products submitted for testing by Qihoo had one of the product's four available antimalware engines, provided by Bitdefender, enabled by default, while a second, Qihoo's own QVM engine, was never enabled. That differed in the product that Qihoo released to the public. In that, the firms claimed, the Bitdefender engine was disabled and the QVM engine active.

"According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives," the firms claimed.

Qihoo, for its part, reportedly said the tests were outdated, too one-dimensional and do not reflect the different online conditions and behaviors in different countries and regions.

At the time, Qihoo accused Baidu and Tencent of similar practices. The three antimalware firms said they found "some unexpected flags within [Baidu and Tencent] products, marked with the names of several test labs and implying some difference in product behavior depending on the environment they were run in."

However, only Tencent has been found to be gaming the antimalware tests, according to the antimalware firms.

Virus Bulletin noted that "these optimizations, which have been found in all recent public versions of the products, provide minimal benefit to normal users and could even degrade the level of protection offered by the products," the company posted to its Facebook page, specifically referring to Tencent.

All three labs will be "imposing stricter controls on participants to reduce opportunities for such actions" in the future, they said.

Join the CSO newsletter!

Error: Please check your email address.

Tags BullsecurityTencentanti-malwaresecurity softwareantivirus

More about AVCFacebook

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Mark Hachman

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts