USENIX: Researchers propose security that adapts to combat malware that morphs

Attackers launch malware that automatically alters itself to avoid detection, and they constantly create new domains where their command-and-control servers can hide, but researchers have come up with security software that detects the presence of attack code even if it has morphed and tracks down domains that infected client machines report to.

A platform called ExecScent can detect C&C traffic and ID the malware family involved or even the specific criminal group behind it, a team of researchers will tell USENIX Security 2013 this week.

[MORE AT USENIX:New security scheme whacks text spammers in hours]

The software carries out a two-pronged attack on malware. First it defines protocol characteristics of HTTP/HTTPS requests between infected clients and C&C servers as a way to recognize malicious traffic, its source and its destination, say the researchers. They are led by Terry Nelms, a Ph.D student at Georgia Institute of Technology and also director of research at Damballa, Inc.

Damballa has incorporated ExecScent into its Failsafe network security appliance as a feature called Request Profiler. Failsafe is usually attached to a TAP port on an Internet router and sometimes to a switch port in order to monitor traffic.

The protocol characteristics ExecScent examines are rolled up into control protocol templates (CPT) for each known type of malware. When actual communications traffic matches a CPT, it is considered suspect. The templates include the family of malware involved and, if available, the criminal groups behind the malware.

Often when attackers craft morphing malware they alter the code to avoid being detected by signature based defenses. But the devices involved in the attacks still have to communicate, and the means they use can be used to identify them, the researchers say.

The second prong of their approach is designed to reduce false positives that may result from matching traffic to a CPT. ExecScent tunes itself to each network it runs on, figuring out what potentially suspect traffic components are actually legitimate. So a particular HTTP user agent header may be commonly used on a given network and so would carry less weight as a factor in determining whether the traffic it enables is malicious.

[HELP:15 (FREE!) security tools you should try]

The value of these components as determining factors varies network to network. For example, a domestic bank network might never send legitimate requests to China, so request to URLs in China would carry a heavy weight in determining that such requests are to C&C servers. On the other hand, a manufacturer with suppliers in China would generate HTTP requests to China URLs frequently, lowering the significance of such requests as a determining factor, the researchers say.

This evaluation and weighting as well as blocking the illicit traffic is done automatically via TCP resets, Nelms says. The system analyzes enough traffic over the course of two or three business days to adjust to each network, he says.

Users can set a detection threshold on the evaluation, that, if exceeded will mean traffic is deemed a C&C request. The higher the threshold, the fewer false positives are returned, but at the same time fewer new C&C domains are caught and fewer infected hosts are discovered, he says.

For the research Nelms and his team are reporting on, prototype of ExecScent was deployed in three large networks for two weeks. Compared to an alternative method of using a domain black list to find command and control servers, ExecScent found more new C&C servers as well as hundreds of previously undetected infected machines.

Nelms says Damballa is working on improvements to Request Profiler that will keep false positives lower and at the same time increase the number of C&C domains and infected client machines that it detects.

If the product generates a false positive, it has to be discovered by end users who can't reach a legitimate domain or by security techs who recognize a blocked domain as legitimate. Then customers have to get Damballa to resolve it, he says.

Tim Greene covers Microsoft and unified communications for Network World and writes the  Mostly Microsoft blog. Reach him at and follow him on Twitter@Tim_Greene.

Read more about wide area network in Network World's Wide Area Network section.

Join the CSO newsletter!

Error: Please check your email address.

Tags USENIXmalware detectionGeorgia Institute of TechnologysecurityDamballaExecScentanti-malwareWide Area Network

More about Georgia Institute of TechnologyInc.MicrosoftTechnology

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Tim Greene

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts