Is penetration testing still effective?

This article was inspired by a mix of recent events and a discussion with a client that asked the question ‘has testing now become so artificial that it is no longer real?’ Having recently started a new business and witnessed the evolution of the security market in Sydney, I find myself as a seasoned tester questioning the value of penetration testing in its current form.

I had to reluctantly agree that the artificiality of a lot of testing has come from the evolution (or devolution) of the testing process. What was once a bespoke field has now become a necessity for a lot of organisations. However, its success and growth has also led a fall in quality.

I see six common faults on the ground:

1. Insufficient scope or context

Hackers without a cause no longer exist, and security vulnerabilities nowadays demand much more context. If the scope of an attack is not understood, if a reason for targeting an organisation is not obvious, or if the broader landscape is unknown, then all you’re doing is spitting out an automated test or vulnerability scan.

Businesses should understand that they, not their technologies or applications, are the targets and testers should understand the importance of reconnaissance in planning its role in facilitating this fact.

Taking the time to identify flaws in business logic will be more rewarding for an organisation and help drive future security planning. In one case with a customer a half hour scoping activity identified IP address ranges that they were unaware of, as well as a misunderstanding of patch management based on what was observed within these ranges. If nothing else, scope and context will identify the most rewarding opportunities for testing in an already constricted time period. I will be writing a future article on how to scope properly and what the benefits are (watch this space).

2. Poor time/space analysis

Anyone who says they can complete testing in less than eight hours at a premium rate may be wasting a business’s time. A lot of penetration testing firms will bank on short, high intensity work with full utilisation periods in order to generate revenue. While all this approach typically needs is a single flaw or a caveat that testing was time-boxed, the reality is that by taking their time a tester will be more conversant with the environment and the organisation, and subsequently provide value for money.

Unfortunately, cost benefit analysis all too often falls back on cost. Businesses should evaluate testing relative to what they want to achieve and how. Testers should have a plan that provides confidence in the assurance process, confirms that the time taken is appropriate, and achieves its purpose with an economy of effort.

Sitting down for a brief planning session, will save time and help shape both time and targets which will make for an effective plan.

3. Understanding qualification, skill and aptitude

This has long been a contentious item within the security community; how do we assure we’re providing appropriate talent and vetting ourselves without providing a false sense of security? Academic qualifications have always represented a baseline, as have some industry certifications. Having said this, customers may not fully understand a qualification or its limitations. This is a greater concern for the security community than the need for its members to achieve qualifications.

Skill and aptitude can be very difficult to measure, however having a standard does help. The Certified Registered Ethical Security Tester (CREST) has helped evolve the measurement of evaluating skills and I think that this will play more in our future evaluation of individuals as we start to move away from quantifiable measurement. Hackers qualify their skills with little more than results and I can only hope such regimes bring about in this approach. In the interim, I personally am looking at people at a grass roots level; are they getting involved and do they love what they do? I would encourage providers & consumers to seek this in their testers; whilst it may not be quantifiable the end result is a more well rounded tester who understands their role in the security assurance process.

4. Effective conduct of testing

A sense of structure goes a long way when testing is performed. Up to this point, good strong lead-in tasks should have set the testers up for success. The following activities indicate effective testing:

  • Keeping the customer informed of start/stop times and updates as to what you’re doing. It may sound counterintuitive to testing and ‘keeping it real’, but it’s surprising how the flow of information keeps it effective.
  • Well documented notes. While I still use pen and paper I had, in previous organisations, used note templates to provide hints and reminders of what I should be doing. In order to have the ‘offensive’ mind ticking over, notes taken against categories or attacks will help provide a clear picture.
  • Cool, calm and measured. Hitting something with a tool or automation triggers alerts and can get people upset. At best you’ve demonstrated that defensive technologies will detect and react against common platforms. A  tester who really wants to be effective should be able to get through with little or no detection or be able to identify where a defender needs to ‘plug a gap’.
  • Ongoing appreciation ‘on the ground’. All plans work until the first shot is fired. What happens if i trigger a denial of service condition? Who do you go to if there’s evidence of prior compromise? How do you act if what you’ve found is inconsistent with what’s been detailed by the customer? What if the environment suddenly blows out into something larger than anticipated? These are things that need to be thought through before and during testing and testers on the ground need to understand this. 

5. Effective outputs of testing

A 400 page report may pass the weight test for a lot of bureaucrats, but will this achieve the result that’s intended? Emphasis on reporting has been a distraction − where an engagement demands an extended report, that’s less time spent on identifying issues and providing practical assistance. I’m a firm believer that shortened reports, video and working alongside stakeholders will ensure that testing is effective.

Procurers of testing need to understand what they want. If it is a report to meet ‘XYZ’ compliance, go for it but sometimes you can save on reporting costs if it’s not what is needed.

Testing firms should have a diverse set of report offerings; if you’ve got a generic template that hardly changes (and regularly features other customers’ names) then you’re doing it wrong.

Human interaction − sitting down with other technically oriented individuals and talking through − will resolve more current future vulnerabilities than a one way paper exchange.

6. The self fulfilling prophecy

My final point and perhaps the most frustrating one is the use of penetration testing as a gateway to more services or specific products. Nothing will undermine the effectiveness of testing (at the individual level and in the community) than the absence of independence.

Before engaging an organisation for testing, customers should understand who the organisation is. Should it happen to be a systems integrator, they should be prepared for the fact that they might be paying for a means to an end.

Validation of independence can only come with time. If you offer a good, objective service that is not used to generate sales, people will keep coming back.

Penetration testing is falling into the same hole as the antivirus industry. Vulnerability assessment services attempt to identify a known attack vector using a known signature which will equate to exploitation. This has become a substitute for penetration testing for a lot of organisations because of its price and ease of use. While penetration testing is at a crossroads I can only foresee that astute consumers will hold it to account, and that providers will adapt or disappear.

About the author

Edward Farrell is a seasoned penetration tester and information security consultant with nearly 10 years’ experience. In 2015 Edward sought to go out on his own and created Mercury Information Security Services. Edward’s new organisation provides customised information security services and advice for Australian businesses.

This article is brought to you by Enex TestLab, content directors for CSO Australia.

Join the CSO newsletter!

Error: Please check your email address.

Tags Enex TestLabOpinionsIT Securityvulnerability scanningCSO Australia

More about CSOEnex TestLab

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Edward Farrell

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts

Market Place