Penetration Testing — Achieving Better Outcomes

The Requirement for Assurance

All networks and applications in organisations are susceptible to attack — from malicious outsiders as well as insiders.

Organisations need to ensure their information security management is robust enough to meet the ongoing demands of compliance and regulation. Rather than looking at systems in isolation, organisations need to adopt a risk and data-centric approach to security.

In particular, it should be clearly determined what needs to be protected, and the business implications of a security breach. Many organisations still struggle, or fail, to identify their sensitive data, where it is stored and who has access to it. This can lead to ill-informed investment in security.

There are numerous types of assessments that can be used to gain assurance of an organisation’s security posture. Penetration testing is a commonly used service, however the effectiveness and value of the results vary widely due to number of factors, such as limitations and constraints of testing, capability of the service provider and the quality of the deliverables. This article provides an overview of penetration testing and a few points to consider in achieving better outcomes.

Penetration Testing

Penetration Testing can be used to determine the current technical security posture of the organisation. It evaluates the effectiveness of protective controls, which can be used to determine the effectiveness of detective and responsive controls.

Penetration testing can also identify opportunities for improvement in information security governance. For example, the findings may identify areas in application development where security was lacking, or where inadequate diligence in change management resulted in insecure settings being enabled on systems - leading to a compromise.

Know Your Data, Define Your Scope

Before any penetration testing commences, the organisation should determine what is being protected. This requires a risk-based approach whereby information assets are identified and the implications of a breach are ascertained. Unfortunately, many organisations don’t know what their sensitive data is, where it is, who has access to it or how many interfaces and access points there are to it. A risk-based approach will identify the scope of testing - breadth and depth. The breadth of scope will define the number of systems, technologies or techniques to be assessed. The depth of scope will determine the number of attributes or complexity to review.


It’s important to remember penetration testing is a point-in-time exercise. Results will only reflect the system at the time it was evaluated.

The environment against which penetration testing is conducted is also important. Organisations frequently elect to run penetration testing against a test environment rather than their production environment for fear of disrupting normal business processes. However, any inconsistencies between them may result in issues going unnoticed when the system is ultimately put in production. Hackers are more likely to focus attention on systems where valuable data can be compromised, therefore assurance of production systems is recommended. Organisations that have a Business Continuity Plan (BCP) or a Major Incident Process should also be able to conduct more thorough testing, knowing they have the capability to deal with any issues that may result.

Once testing is completed and a report issued, the value of testing can only be realised if identified issues are remediated. While it may be difficult to predict how much time and money should be allowed for remedial activities before the test, there needs to be an allocation made, with provision to seek further resources (personnel and monetary) if warranted.

One of the most common limitations of penetration testing is when a test is simply run to address an audit or compliance requirement. This is normally associated with a compromised procurement process, the selection of low-cost (less comprehensive) suppliers/resources, and less interest in the result, than passing an audit. This is an abuse of the true intention of a penetration test, which is to identify the issues that expose an organisation to risk, including those that should be addressed for regulation and compliance. This is commonly called “check box testing” or “security for compliance sake”.


The selection of the right service provider to conduct testing is essential to an effective outcome. A service provider’s capability should be assessed with respect to the following attributes: methodology, experience, skill and tools. All of these competencies need to be present for a service provider to deliver effective outcomes.

Testing can only be considered comprehensive if conducted in accordance with an appropriate methodology. Just because a company performed a network penetration test doesn’t mean it can deliver wireless or mobile application assessments. Information Security is a specialist field and each domain of testing is considered a distinct specialist discipline. Service providers should be able to demonstrate the substance behind their service methodology.

Experience comes from maturity and effective capability in penetration testing cannot be developed overnight. Companies with heritage and specialisation in penetration testing can easily demonstrate a track record and depth-of-knowledge. They will have published whitepapers, demonstrated relevant security advisory services and make a contribution to the security community. The capability of the tester – the individual - needs to match the capability of the service provider to deliver a quality outcome. A tester needs to deliver with flair, but also follow the defined processes and methodology accurately.

The innate capability of a tester to deconstruct a system will be demonstrated through a combination of knowledge and tools. Tools assist testers through automation, using the processing power of the computer combined with purpose- developed software to systematically probe your systems to identify points of weakness or exposure. However, there is more to penetration testing than simply spraying a network with a myriad of noisy scanning tools, hoping it will do the job for you. A professional approach interprets scanned results, accurately exploiting vulnerabilities with finesse. Tools may provide insight into technical flaws but they cannot identify logic flaws. For example, a technical flaw could identifying a system vulnerable to a buffer overflow attack; but a logic flaw may determine than an application can be tricked into executing something outside of its intended business purpose (such as refunding a transaction twice, which could be used to defraud a business). It takes intellect and finesse to identify logic flaws. Advanced testers are likely to write their own tools to perform precise actions which complement the range of the commonly available products.


A report is the tangible product of a penetration test so the ultimate impact of the project will rely heavily on the quality of the report. The following attributes should be sought from the deliverable:

  • Ease of use: Simply put, reports must be easy to use. Otherwise they become paperweights or put in the too-hard basket.
  • Accuracy of findings: All findings must be validated; there should be no false positives reported.
  • Action for remediation: Technical findings should be concise but precise. There should be adequate detail for the recipient to remediate issues with no further research. This really comes down to the capability of the supplier to produce a quality deliverable.
  • Technical and business risk: The report must articulate both technical and business risk. This requires a deeper understanding of the organisation, the market and compliance and regulatory requirements. The executive summary must have adequate information so that it can be used as a business case for further investment (e.g. funding for remedial efforts, hardware, software etc). In general, the report must provide information that can be communicated upwards in an organisation so that business executives can digest the contents and understand the implications of technical risks.
  • Anatomy of attack: Where a system is compromised, the exact steps, replete with screen shots, should be provided to allow the organisation to reproduce the exploit. This will allow the organisation to ascertain the implication of the compromise if the testing was time constrained and no further analysis conducted.
  • Root cause analysis: If a recommendation is limited to cosmetically addressing an issue, it is likely the issue will re-emerge some time later (probably with greater significance and impact). The root cause of the issue needs to be identified. This takes further time in research and analysis but undoubtedly provides better value in the outcome.

Where does Penetration Testing Fit In?

It’s important to realise penetration testing will not be effective if conducted as an isolated exercise. Testing should be part of an ongoing information security management lifecycle (Plan, Do, Check, Act). Testing should be conducted in the plan and check phases, and issues remediated in the do and act phases.

Industry standards define the testing requirements (eg. OWASP). Testing will verify the implementation of security requirements (eg. the organisation’s development, configuration or deployment standards). Governance Standards (ISM, ISO 27001) will include the organisations security requirements and require regular testing to validate the security of the process. Outcomes of the tests will identify risks that must be addressed through the organisation’s Risk Management capability.

The Cloud

Many organisations are moving to the cloud. Organisations are expected (and mandated by government and industry regulations) to take reasonable steps to monitor, review and audit information security effectiveness of cloud providers. This can include engaging internal or external auditors and specialist firms to deliver penetration testing because it provides a good technical assessment of the effectiveness of a cloud offering, and should be included as part of a vendor capability review. End users of cloud services are entitled to expect their personal information to be protected from loss, misuse, unauthorised access, modification or disclosure. Test scoping would potentially also include network and application testing against the public facing systems, internal systems, and a review of the ability of the cloud provider to maintain isolation in a multi-tenanted deployment.


Don’t think of your organisation as silos of systems. Adopt a risk-based approach and understand what it is that you need to protect and the implications of a security breach. Know your data. Determine where it is, who has access to it and how many systems or interfaces there are to it. Only then can you determine how you are going to test the effectiveness of the controls protecting it. Your scope should be determined through risk assessment.

Due to the increased sophistication of attacks there is an increased requirement for expertise. Organisations will need to rely less on tools and more on expertise to interpret and evaluate their security posture.

Assessing the capability of a service provider is essential in making an informed decision when selecting a party to deliver penetration testing. Capability is a function of methodology, skill, heritage and reporting. Quality of reporting should be evaluated because it affects the ability to remediate.

Penetration testing in isolation is of limited value. Testing must be part of a broader information security management capability, delivered through a vulnerability management program. Having a test conducted and getting results is the initial phase of an assessment program. You also need to be able to act on the recommendations which will require planning, resources and budget.

Price is important. But consider also that comprehensive testing will cost more than check box testing and testing in breadth and depth will incur greater costs – but also deliver more value. With organisations electing to use cloud services, as extension of your organisation, its inherent risks must be evaluated and treated accordingly. A penetration test should be treated as an opportunity to evaluate and tune your detection and response capability.

Murray Goldschmidt, Chief Operating Officer,

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Have an opinion on security? Want to have your articles published on CSO? Please contact CSO Content Manager for our guidelines.

More about ISMISO

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Murray Goldschmidt

Latest Videos

More videos

Blog Posts