Security Manager's Journal: Virtual machines, real mess

When Internet and phone service goes down, the problem is traced to VM images installed in a classroom.

It started out as a simple call to the help desk from an engineer at one of our major development centers: Phone calls were being dropped. Soon, similar complaints were coming in from other engineers, as well as from sales associates, who said the inability to maintain phone calls was making it difficult to close deals.


Trouble Ticket

At issue: Phone and Internet service is severely compromised at a development center.

Action plan: Restore service, then find out what went wrong. And once that's done, take steps to avoid a repeat.


Anything that affects revenue is sure to get someone's attention. The telecom team checked out the Cisco call manager and gateways; they were fine. It wasn't until the help desk received a new set of complaints about Internet connectivity being slow at that same development center that someone decided to get the security department involved.

The head of our network team, who is also responsible for firewall administration, sent me a message that was sure to get my attention: "You better come check this out." What he had to show me was that the logs from the firewall protecting the development center were filled with outbound connections over Port 445 to several locations on the Internet.

We had to contain that activity quickly to return Internet and phone service. Our attempt to block the outbound traffic at the firewall didn't succeed, since the logs had taken up so much of the firewall's resources that we couldn't do anything at all on the firewall. The network engineer placed an access control list on one of the routers, which eventually allowed him to modify the firewall rule to block the bad traffic. That got us back the Internet and phone service, so the immediate problem had been remediated. But what had caused it? I had the engineer back up the logs so we could analyze the data.

Our review showed that the IP addresses that were generating the traffic were assigned to a classroom. The instructor told me that the trainees had installed a virtual server image on the classroom desktops and, contrary to normal classroom protocol, connected the virtual machines to the corporate network. We found that those virtual machines were not running any antivirus software and hadn't been patched in more than two years, so we ran a virus scan of one of the virtual machines. Suddenly, everything became very clear.

The virtual machine was infected with a virus whose characteristics matched the activity that caused the denial of service to the office. In fact, all 30 desktops in the classroom were infected. But that's not the worst of it.

The installed images were derived from a base image maintained at a cloud provider. That base image contained the virus, which explains how 30 machines became infected.


I then moved on to the person who was responsible for provisioning virtual-machine images to find out why steps hadn't been taken to avoid an infection. He explained that a couple of years ago some patches had caused images to become unstable, so patching was stopped. As for antivirus software, he said he didn't have the budget to install it on more than 1,500 Microsoft Windows images. Perhaps that explanation was supposed to mollify me, but I could barely contain my dismay. Fifteen hundred VM images that had little or no protection from viral infection! And those images were regularly used by several departments on machines operating on our corporate network.

I immediately called a meeting with our CIO and the vice presidents for the divisions that deploy virtual machines. I called for an immediate mandate to scan all images, install our corporate antivirus software, update all patches and put a process in place to ensure that images comply with the company's patch management process.

All in a day's work, right?

This week's journal is written by a real security manager, "Mathias Thurman," whose name and employer have been disguised for obvious reasons. Contact him at

Join the CSO newsletter!

Error: Please check your email address.

Tags securitycisco

More about ----CiscoMicrosoft

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Mathias Thurman

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts