What is phishing success?

A recent article asking the question to security professionals seemed to miss the mark, and raises more questions than it answers.

In putting together awareness programs for dozens of clients, the potential to integrate phishing simulations always comes up. For the most part, it seems like a staple of awareness programs. But when the concept of phishing is raised, I always ask, “Why?”

Yes, the question potentially costs me money. Also while most people perceive the phishing simulations as a direct way to decrease phishing susceptibility, the decrease might not be relevant or significant. So when I looked at a recent CSO article that asked security experts what they thought “success” meant when it came to phishing simulations, I was a frustrated.

The comments from security experts mostly focused on a reduction in clicking on simulated phishing messages. I assume people believe that if fewer people click on a simulated phishing message, fewer people will click on a real message. That is not necessarily the case. This discussion is actually much more complicated than it appears, and it involves dispelling many myths and specious beliefs about phishing.

What is security success?

Before looking at success in phishing simulations, we must first consider what is success for overall security efforts. First off, there is no such thing as security. The dictionary defines security as freedom from risk. There will always be risk, so security is unattainable. An implementable definition of security is risk management.

Risk management is essentially the act of cost effectively mitigating loss. In short, security efforts are successful if you reduce your loss by more money than your security countermeasures cost. For example, if you invest $500,000 in anti-malware software, and you reduce the costs of loss due to malware by more than $500,000, your security program is successful. If you reduce loss by less than $500,000, your program, or at least anti-malware, failed.

There is a general problem with this measure, as most organizations do not adequately track security-related losses. Without the appropriate metrics, it is hard to prove success. However, the principle is straightforward. If you plan in advance, you should at least attempt to gather the appropriate metrics.

The problems with phishing simulations

There are several critical issues with implementing phishing simulations. The first one is the actual receipt of the messages. With all services, you have to white list the messages to ensure they get to the recipients. So, you are testing people with phishing messages that they would never receive, as the white listing is implemented to avoid the messages getting sent to spam files or from being deleted, before reaching the recipients.

[ MORE ON CSO: How to avoid phishing attacks ]

Then there is the fact that just because a user does not click on one phishing message, it doesn’t mean they will not click on others. Some people might not click on cat videos, while they would click on a shipping message.

Then there is the sophistication of phishing messages to consider. I can purposefully manipulate the user response rate, if I choose. For example, if I want to show success in the program, I can create a very sophisticated message that uses inside information and is related to some timely event, and get a very high response rate. I would then follow it up with a more generic phishing message, such as a shipping message with poor grammar, and would get a very low rate.

The referenced article states that if phishing simulations get a 10 percent response rate, the effort is a success. As the previous paragraph highlights, a 10 percent response rate can mean little in actual effectiveness, depending upon the simulated phishing message used. However, even if you assume it is the most sophisticated simulated phishing message ever, that means that a significant number of people within an organization will still respond to the message.

More frequently, users begin to recognize the simulated phishing messages and do not respond, not because they are more aware of phishing concerns, but because they are aware of the simulations. Another common occurrence is that if one person detects a phishing message in an organization, they may then warn their coworkers about the message. The coworkers will then know to proactively delete the messages. In more than one simulation I was involved in, companies proactively warned employees that they will receive a simulated phishing message within a given time period for political reasons.

Phishing messages require technical failures to be successful

While security professionals seem to attribute responses to phishing messages as a demonstration of poor security awareness, it is actually a much more complicated issue. Again, there had to be a technical failure for messages to get to the user. More important, just because a user responds to a message, it does not mean that there should actually be a loss.

Join the CSO newsletter!

Error: Please check your email address.

More about ClickCSO

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Ira Winkler

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts

Market Place