Measuring the effectiveness of your security awareness program

Organizations that maximize the efficacy of their security awareness programs may receive many benefits

As Yogi Berra put it, "If you don't know where you're going, you'll end up someplace else." Do you know where you're going with respect to your privacy and security awareness programs? How will you know when--or if--you get there?

[How to use Syrian Electronic Army attacks to improve security awareness]

"But wait just a minute," you object. "Everyone knows that security is a process, not a destination. Is there really any such thing as arriving?" Well, of course there is. Just because a process is dynamic doesn't mean it's left without any measurable aspects. Besides, if any process is to be improved, it must also be measured.

There are many benefits an organization will enjoy when it makes those improvements, not the least of which is the budget justification for creating a security awareness program that help will boost security effectiveness overall. Martin Sadler, Director of Security at HP Labs, summed them up thusly: "Organizations that have achieved a high level of security effectiveness are better able to identify major data breaches, secure confidential information, limit physical access to data storage devices, and achieve compliance with legal and self-regulatory frameworks. They are also in a better position to attract and retain high-quality security personnel and enforce corporate policies."

Those benefits have ripple effects throughout the organization--benefits that span protecting the company reputation to increasing customer trust and loyalty. And those translate directly to the bottom line.

Granted, measuring security effectiveness is not as straightforward as measuring a manufacturing process. There are many variables that are simply outside of one's direct control. In fact, a recent ISACA report conceded, " is contextual and not an isolated discipline; it depends on the organization and its operations. Furthermore, effective security must take into account the dynamically changing risk environment within which most organizations are expected to survive and thrive." All the more reason that improvements be addressed wherever possible!

In any case, this variability may explain the disparity of results Dr. Kenneth Knapp discovered when he investigated the effectiveness of security programs. He found that while the majority of infosec professionals surveyed believed they were able to secure their information effectively, only 22 percent of them believed so with a high degree of confidence.

[Punish careless employees to reduce security breaches, vendor says]

Moreover, the survey showed that more than a third did not believe that their organization effectively secures its data. And this is likely understated. Sounds like room for improvement.

When asked about this, Dr. Larry Ponemon of the Ponemon Institute admits that while security effectiveness can be an elusive object to measure, there are highly effective ways of determining it, short of recording incidents of catastrophic failure. So how do we go about making improvements? What, exactly, is it we can measure to determine whether the security awareness program is as effective as it ought to be? In answering these questions, Ponemon begins with identifying the key dimensions of information security effectiveness, which he describes as:

  • Uptime: The ability to withstand cyber attacks and avoid costly business disruption.
  • Compliance: The ability to achieve compliance with all applicable regulations and laws.
  • Threat containment: The ability to prevent or quickly detect external security threats such as cybercrime, social engineering or malicious attacks.
  • Cost efficiency: The ability to manage investments in information security and data protection in a competent (non-wasteful) manner.
  • Data breach prevention: The ability to prevent or quickly detect internal security threats such as the negligent or incompetent insider.
  • Policy enforcement: The ability to monitor and strictly enforce compliance with internal policies, procedures and other security requirements.

These are the metrics Ponemon applied when he developed his breakthrough Security Effectiveness Score, which, in its most compact version, evaluates 24 attributes (extrapolated from the six key dimensions described above) that consistently correlate with strong security postures. In short, the higher the score, the stronger the organization's security posture, the greater its ability to avoid a breach, and the lower the cost to mitigate a breach. In other words, an objective standard of measure for security effectiveness.

One of the most significant insights that resulted from the application of the tool is that of the 24 parameters considered, 75 percent of them are directly related to security-aware behaviors, not just information technology. And when specific employee behaviors are addressed in a meaningful way to bring about a security-aware culture, the incidence and cost of non-compliance plummets.

[How to create security awareness with incentives]

More importantly, when correlating the scores to actual breach incidents, Ponemon's data (gleaned from the more than 7,000 security effectiveness score surveys he's collected) also demonstrates that when organizations spend a dollar on information security--and particularly on security awareness training--they get far more than a dollar's value in return. In other words, an ROI. Another reason to see how your organization measures up!

Lastly, ISACA points out in "Security Awareness: Best Practices to Secure Your Enterprise" that measurement not only reveals whether the awareness program is effective, but it can also help identify any knowledge gaps and ensure the improvement of the program overall. Surveys, interviews, pop quizzes, exams, and audits are a few of the more common assessment tools that can be used to measure progress.

A case in point is Western Union's approach to measuring the results of its security awareness program. Western Union's Kim Hickman explains, "Of course you always wonder if you're making an impact, if your efforts are paying off. So to gauge and quantify that we started conducting 20-question quizzes, sent to a different sampling of the employee population every month. We trend the scores over time to see if, as an organization, we're getting better. And we have seen improvement since we launched our new security course, with quiz scores now averaging 89%. It is definitely raising awareness and changing behavior."

Furthermore, Hickman also observes that the quizzes have the additional effect of reinforcing the security awareness information presented in their course. "We get a double benefit there," she says.

The bottom line? The very act of measuring actually also helps bring about the desired result!

John Schroeter is a security awareness programs strategist.

Join the CSO newsletter!

Error: Please check your email address.

Tags security

More about HPISACAWestern Union

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by John Schroeter

Latest Videos

  • 150x50

    CSO Webinar: Will your data protection strategy be enough when disaster strikes?

    Speakers: - Paul O’Connor, Engagement leader - Performance Audit Group, Victorian Auditor-General’s Office (VAGO) - Nigel Phair, Managing Director, Centre for Internet Safety - Joshua Stenhouse, Technical Evangelist, Zerto - Anthony Caruana, CSO MC & Moderator

    Play Video

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

More videos

Blog Posts

Market Place