Tuning the security analysts

Monitoring of the SIEM has gone offshore, but the overseas analysts are escalating a lot of events that aren't really worth investigating.

I have told you about the need to tune our security incident and event monitoring (SIEM) tool. Now we need to tune the analysts who are monitoring the SIEM.

We had no problem when SIEM monitoring was done in-house. My security team consists of two Level 3 analysts, who are well seasoned and very familiar with our company and have more than seven years of experience in information security. But it was recently decided that SIEM monitoring was something we could offshore. There's no arguing with a decision like that. The cost of offshore services such as this is compellingly competitive, and we've had good experiences with our offshoring of the help desk, network operations and development. Besides, my team wouldn't be laid off but instead freed to do more pressing things.

And so we found ourselves executing the statement of work and creating dozens of operating procedures, an escalation matrix, incident response training materials and a fairly comprehensive incident analysis reporting tool. We conducted countless interviews and endured weeks of knowledge transfers, several overseas trips, negotiations, dog and pony shows and other activities, bringing us to where we are now: disappointed.

The thing about SIEM analysts is that they need to be judicious. Hundreds of devices report their logs to the SIEM, which currently is identifying about 1 billion events per quarter. Clearly, we can't investigate them all, and in fact we average about 1,500 events of interest per quarter. We do far fewer investigations than that, though, since some events that wouldn't be worthy of investigation on their own become of interest when they are correlated with others to show a pattern. But when you're dealing with a billion events every quarter, managing false positives is essential, and that's why it's necessary to properly tune both the SIEM and the analysts.

For example, like most companies, mine is constantly getting scanned. Many years ago, when port scanning was less common, a single incident might have warranted action. Today, though, if we responded to every port scan, we would need an army of analysts. Therefore, we've created rules to trigger only when a port scan exceeds certain thresholds, suggesting that someone is being very persistent, versus just "driving by." So when one of the new offshore analysts began escalating dozens of port scans that didn't meet our pre-defined criteria, we had to spend time tuning the analyst.

Another example had to do with one of the analysts responding to a single FTP "change working directory" alert. This event triggers when someone who has accessed an FTP server attempts to enter more than a certain number of characters when changing to a different working directory. Such activity can be a prelude to a denial-of-service or buffer overflow attack. However, none of our FTP servers are vulnerable to attacks related to misuse of the FTP CWD command. The analyst should have made sure of this before escalating the event.

The problem with false positives, of course, is that my staff, instead of having extra time to respond to real events now that they have been freed from SIEM monitoring, are losing time by having to look into events that don't warrant the attention. And some of these ludicrous security incidents are forwarded to IT operations team members for action. If they see too many of these false positives, they could end up with "the boy who cried wolf" syndrome and begin ignoring incidents. Besides all that, false positives are tracked; when you have a lot of them, management receives the wrong message.

All we can do, it seems, is to hold lots of training sessions and reduce the number of false positives coming from overseas. Just as important, of course, is the need to do that while making sure that the analysts don't miss the events that really do deserve our attention. When you think about it, going too far the other way would be even worse.

This week's journal is written by a real security manager, "Mathias Thurman," whose name and employer have been disguised for obvious reasons. Contact him at mathias_thurman@yahoo.com.

Join in the discussions about security!computerworld.com/blogs/security

Join the CSO newsletter!

Error: Please check your email address.

Tags securityIT outsourcingSIEM

More about

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by By Mathias Thurman

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts