Palo Alto next-gen firewall marked a 'caution' in NSS Labs test

Next-gen firewall pioneer falls down on evasion tests

Do independent security tests make or break products? Normally not but the latest assessment of next-generation firewalls (NGFWs) by testing firm NSS Labs will make mixed reading for some of the products with one - Palo Alto's PA-3020 - described as "below average" with a rating of "caution".

Most of the other 11 NGFWs in the test performed pretty well on NSS Labs' Security Value Map (SVM), which with its X and Y axes vaguely resembles a sort of Gartner Quadrant for security systems.

Eight out of twelve of the systems received a 'recommended' status, exceeding 90 percent for security effectiveness - Check Point's 13500, Cisco's 5525-X and Firepower 8350, Dell SonicWALL's SuperMassive E10800, Fortinet's FortiGate-1500D and FortiGate-3600C, McAfee's NGF-1402, and WatchGuard's XTM1525.

Neutral ratings were awarded to Barracuda's F800b, Cisco's 5585-X SSP60 and Cyberoam's CR2500iNG-XP for a variety of reasons varying from below average security effectiveness (Barracuda and Cyberoam) to price-performance (Cisco). Palo Alto's PA-3020 was marked 'below average' on both security effectiveness and price-performance.

The positive news anyone looking to buy one of these products is that the total cost of ownership (measured against Mbps protected) is half what it was in 2013 at only $21.80 - high-end firewalls are getting cheaper for a given throughput.

But some failings are still apparent.

"Evasions continue to be a challenge for the industry," said NSS Labs CEO, Vikram Phatak. "To date, every single NGFW group test has resulted in at least one vendor missing one or more critical evasions.

"If someone uses an evasion to circumvent a security product, you will never know until you are compromised. This is why ongoing independent testing is so important to cyber resiliency," he said.

So what went wrong for Palo Alto's system? The PA-3020 blocked 93.1 percent of attacks against server applications, 92 percent of attacks against clients, giving a 92.5 percent overall score, which sounds quite good. What let it down was its ability to protect against three classes of what are called 'evasions', techniques for disguising an attack to avoid detection.

As NSS Labs states: "many of the techniques used in this test have been widely known for years and should be considered minimum requirements for the NGFW product category."

Techworld received no statement from Palo Alto by press time but in comments to another website the company questioned the NSS Labs methodology.

Although the firm will receive negative headlines for the latest test, in the past NSS Labs has given its products passing marks while calling out other products. As with any class of security product, performance varies over time. The fact that Palo Alto didn't do as well as its rivals in this test doesn't mean that it will do so in the same test in a year's time.

NSS Labs has been here before with other vendors. Earlier this year it dished out some pain to FireEye's Web MPS 4310 and Email MPS 5300 products which were given the same 'caution rating. That generated some heat as FireEye sought to defend its products in public. Perhaps wisely, Palo Alto has yet to copy that tactic.

Join the CSO newsletter!

Error: Please check your email address.

Tags NSS LabswatchguardFortinetNetworkingSSPcyberoamiGateM1DellGartnermcafeesecurity

More about CyberoamDellDell SonicWALLFireEyeFortinetGartner

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by John E. Dunn

Latest Videos

  • 150x50

    CSO Webinar: Will your data protection strategy be enough when disaster strikes?

    Speakers: - Paul O’Connor, Engagement leader - Performance Audit Group, Victorian Auditor-General’s Office (VAGO) - Nigel Phair, Managing Director, Centre for Internet Safety - Joshua Stenhouse, Technical Evangelist, Zerto - Anthony Caruana, CSO MC & Moderator

    Play Video

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

More videos

Blog Posts

Market Place