The team gathered in the security lab for our weekly meeting. At the top of the agenda was distributed intrusion detection. The junior-level security engineer glanced toward a whiteboard that displayed his work of art -- numerous lines drawn in different colors depicting the complicated layout of our network. The black lines represented copper links, the red lines fiber, the black boxes routers and switches, and the blue boxes network taps and Snort sensors.
Meanwhile, the senior security architect distributed current network diagrams neatly done in Visio network diagramming software. Everyone was well prepared.
I noticed, however, that the team members were on edge. They were feeling drained and discouraged after dealing with Microsoft Corp.'s flurry of security patches in October, discovering a security breach that originated from a remote laptop, managing participation in more than 40 IT and development projects and hearing rumors that the 2005 budget was being slashed to bare bones.
I took a deep breath, smiled my best we're-all-in-this-together smile and turned to the youngest member of our team. "Whatcha got?" I asked him.
He explained that his first recommendation for how to gain better visibility on the network was to buy port aggregators. He had read reviews and thought the idea sounded good. But after working closely with network engineers to better understand the network, the team member had found that the port aggregators wouldn't work for a variety of reasons. So, instead, he had come up with a design that combined port spanning with network taps. I reviewed his design and nodded. This kid is smart, I thought to myself, and this is sure going to cost a lot less money than the original plan of purchasing port aggregators at US$950 a pop.
The goal, in my mind, was to be able to monitor the LAN/WAN environment for suspicious traffic, receive alerts via SMTP (preferably e-mail sent to our BlackBerry devices), respond to events in as close to real time as possible and report on that activity in weekly and monthly summaries to upper management. I didn't think that was too much to ask.
The senior security architect turned to me and said, "It would be helpful for us if you would be very explicit about what kind of results you're looking for."
As usual, I stuck my foot all the way down my throat. I said, "I want to see 100 percent of the traffic on 100 percent of this network and every network attached to it that we have security responsibility for." He smiled gently and said, "That's impossible. And by the way, nobody does that."
I learned early in my career that if you want to manage people well, you have to let go of your ego, hire people a lot smarter than you and let them do their jobs. My job is to understand enough about security to make good business decisions and to ensure that my team members have what they need to do their jobs. So when presented with this bit of insight from the security architect, I laughed, and the team started to loosen up.
Tell me everything
"What I know," I said, "is that we do not have the level of visibility that we need. What I know is that we cannot continue to rely on span ports. What I know is that our network could be owned (that's the hacker term for obtaining undetected full privileges on a private network) right now, and we have to do something. You're the experts. Tell me how this is done; tell me if we can do it and what you need from me to get it done."
At this point, the entire team started joking about how the network architecture is horrible. I thanked our junior engineer and asked him to put his designs into Visio, meet with the network engineers and the senior network architect and gain buy-in for his designs. Without the cooperation of the network guys, we weren't going to be able to sell the idea of in-line network devices, no matter what the cost savings.
I had to put together a project plan for this, present it to management, get it on the project management office queue and juggle the budget to make sure we could move forward quickly. My part is boring but essential. The company was scrutinizing every dollar, and I couldn't let this project fall by the wayside.
In an effort to be explicit about my expectations, I asked the senior security architect to explain the nuts and bolts of security monitoring to the team. I wanted to understand what we could do without seeing all the traffic all the time.
"What we need to be able to do," he explained, "is to collect data that describes the network environment to the highest levels possible. We will be constrained by hardware and its ability to collect the data. Therefore, we will have to focus on sampling the data at key points on our network. We will not be able to collect every packet that traverses the network and analyze it. Even if we did collect gigabytes of data from each key point, we would be faced with correlating and analyzing vast amounts of data, and we just don't have the resources to do that."
We had been in the security lab for two hours discussing intrusion detection, and it was now lunchtime. Engineers aren't at their best when they're hungry. "OK, guys, let's break," I said. "The next step is to find a way to correlate and analyze the data now that the design for collecting it has been completed. I'd like to have some choices to review next week. Good work."
I was humbled by my inability to understand all the technical details, but I was encouraged that I had hired people with top-notch skills. I was also excited that we were finally making some progress at the core level toward finding out what was really going on in the network.
What do you think?