NSS Labs fires back at FireEye as security test controversy burns anew

Replies to FireEye rebuttal over mediocre test results

Security testing firm NSS Labs has publically defended itself against furious accusations by security firm FireEye that a cool assessment of the security vendor's breach detection technology published last week was based on a flawed methodology.

Testing security products is a complex undertaking riven with uncertainties about whether any assessment can possibly simulate real-world attacks, which doesn't, of course, stop security vendors quoting these results when they do well.

Less frequently, when a vendor does badly - or just not as well as its rivals - the fur can start to fly. And so it was when last week NSS Labs' Breach Detection Systems Comparative Analyst Report gave FireEye's Web MPS 4310 and Email MPS 5300 systems a lower rating on its Security Value Map (SVM) compared to equivalent products from SourceFire, Trend Micro, Fortinet and Fidelis.

NSS Labs' assessment could be described as relatively stinging, slapping FireEye's product (and one from South Korean firm AhnLab) with a 'caution' while the others received a' recommended'. Anyone who believes that nobody reads these reports, or that they have little effect, might want to ponder the effect on FireEye's share price, which dropped nearly 8 percent on 3 April (although tech stocks were hit anyway the next day).

This would be a troubling day for any security company but for a firm barely six months on from a high-profile and well-subscribed IPO, any bump is unpleasant. Wounded, FireEye senior vice president Manish Gupta came out swinging, criticising the test methodology on a number of counts, in particular the selection of malware against which systems had been assessed, which he believed skewed FireEye's results down.

He also said the firm had "declined to participate in this test because we believe the NSS methodology is severely flawed," and that the "FireEye product they used was not even fully functional, leveraged an old version of our software and didn't have access to our threat intelligence."

It's a high-risk strategy for FireEye because it draws more attention to the results and risks the firm getting drawn into a verbal exchange that attracts even more rubber-neckers who don't understand the complex issues at hand. Sure enough, NSS Labs has today published its rebuttal of Gupta's claims.

In a post Don't Shoot the Messenger NSS Labs' Bob Walder denied that FireEye had not been a willing participant and said the firm's products were installed and configured by its engineers during 2013. Walder also rebutted Gupta's various claims over the testing methodology in some detail.

"In the grand scheme of things, FireEye's results were not that bad. The real issue here is that FireEye now has credible competition in the BDS [breach detection system] market place and the data from this NSS test shows it," wrote Walder.

That bring us to the really contentious thing about this test - on numbers alone FireEye really didn't do that badly, detecting 95 percent of web malware, 96 percent of email malware and 93 percent of exploits, giving an overall detection rate of 94.5 percent and a zero percent false positive rate. Although this is below the roughly 98-99 percent scores achieved by most of its rivals, the real problem NSS Labs found with the FireEye systems was their cost-performance.

This plots the total cost per Mbps protected against security effectiveness, which in the case of FireEye left its product with a figure of $427.85 (£280) against the highest-rated Sourcefire Advanced Malware protection costed at $231.86. In NSS's assessment at least, Sourcefire simply offers more protection for every dollar spent than does FireEye.

Regardless of the arguments on either side of this judgement, it is clear that breach protection security comes at a premium more or less which company is looked at; these are all expensive systems and measuring value for money and effectiveness will remain a black art shrouded in technical complexity. It is also the case that working how good they are at living up to the claims made in the sales brochures is not going to be as easy in 2014 for any firm as it was a year or two ago.

Join the CSO newsletter!

Error: Please check your email address.

Tags NSS LabsConfiguration / maintenanceFortinettrend microsecurityhardware systemsFireEyeData Centre

More about Cisco SecurityCisco SecurityFireEyeFortinetMessengerTrend Micro Australia

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by John E Dunn

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts