When Richard McKinney set out to migrate the Department of Transportation (DOT) to Microsoft Office 365, he got a valuable lesson in shadow IT, one that could serve as a cautionary tale for other government leaders as they look to upgrade and consolidate their systems.
McKinney, who only recently stepped down as CIO at DOT, had been leading a turnaround mission at the department since his arrival, but when it came time for the Office 365 rollout, he quickly discovered how chaotic the situation was, with hundreds of unauthorized devices running undetected on the sprawling network.
"No one sat down many years ago and designed a network for the Department of Transportation," McKinney tells CIO.com in a recent interview, describing how various outposts in the department's sprawling operations had "stitched together" networking equipment as needs emerged. "We didn't have an overarching, as-is blueprint for the department's network."
So McKinney set out to create one. He hired a vendor called Decisive Communications to comb through the DOT's network and identify the unauthorized devices running in that far-flung environment. Decisive used technology from Riverbed to analyze the network, and quickly found more than 200 previously undetected networking devices, including many that still had factory-issued passwords.
[ Related: CIOs vastly underestimate extent of shadow IT ]
As it turned out, it had not been uncommon for staffers at the various administrative outposts of the Transportation Department to take it upon themselves to beef up networking capacity at the local office. Say a 16-port switch filled up and the office was still adding more staff -- the solution might be to go to Best Buy and buy a new switch to accommodate additional users.
"It was like self-serving, if you will," McKinney says. "They tended to be more like consumer devices," he explains, whereas "we would buy more enterprise-ready equipment."
"That brought us a laundry list of equipment that we needed to replace," McKinney says.
Security and the 'weakest link'
The discovery of all those unauthorized networking devices gave McKinney pause, raising obvious concerns about the security of the Transportation Department's systems. After all, if all those potential entry points were running on the network with no central management or visibility, it wasn't unreasonable to fear that malicious actors might have infiltrated the system. What's more, because of the network's "flat" design -- the product of an ad hoc development with no overarching architecture -- an intrusion into one low-risk corner of the network could afford access to more sensitive, mission-critical areas, McKinney says.
"Once you got on the network it was easy to traverse the network. It wasn't segmented," he says. "I think it drove home the point that we're all in this together and the chain's only as strong as the weakest link."
It was also clear that those security concerns were much bigger than just an IT problem. McKinney says that he initiated a thorough scan of the network and found no evidence that DOT data or systems were compromised, but he also notes that skilled intruders "don't leave a trail of cookies." In any event, McKinney brought his findings to the department's brass.
"I felt like it was potential security vulnerability, and it was my responsibility to tell the political leadership about it," he says. "[It was] not only an eye-opener for us, but also an eye-opener for our departmental leadership."
The shadow IT revelations and the associated security concerns led McKinney to launch a project to rearchitect the DOT's network, an effort that, while still ongoing, has been introducing more centralized controls and clearer segmentation to cordon off the systems of various administrations within the department.
The experience also compelled his office to change the internal processes for introducing new equipment to the network, including a policy directive putting the various DoT administrations on notice that the days of ad hoc, unsecured and unmanaged network expansion were over.
"We also at that point put out policy memos and told the entire department that there will be no adding equipment to the network without going through a formal change-management process," he says. "We had one, but people had been ignoring it."
Now, an alumnus of the federal government, he is preaching a message of network visibility and centralized management.
"I think it's really good to start to make sure you have a clear and complete understanding of your infrastructure and your network, your servers and all your connections to the internet," he says. "I'm a huge proponent of you've got to know what you own, and you've got to manage what you know well."