Out of Control

Industrial control systems sit squarely at the intersection of the digital and physical worlds. They're vulnerable, they're unpatchable, and they're connected to the Internet.

Vitek Boden sought revenge. After he was turned down for a job with the Maroochy Shire Council in Queensland, the 48-year-old disgruntled techie unleashed his anger in early 2000 by hacking into the town's waste-water system at least 46 times. On two separate occasions, his electronic attacks (apparently he used a stolen laptop and a radio transmitter) led to pumping station failures that caused as much as 1 million litres of foul-smelling raw sewage to spill into parks, waterways and the grounds of a tourist resort. In the surrounding area on Australia's Sunshine Coast, creeks turned black.

Boden was a disgruntled ex-employee of Hunter Watertech, the company that had recently installed Maroochy's computerized sewage control system. Boden's attack became the first widely known example of someone maliciously breaking into a control system. But there have been other control system breaches, including, for example, a 1997 control tower shutdown at Worcester Regional Airport in the US state of Massachusetts and a Slammer-related disruption of the safety monitoring system at FirstEnergy's Davis-Besse nuclear plant in Ohio.

Electric utilities, oil and gas refineries, chemical factories and even food processing plants use control systems to digitize and automate tasks once handled by people: opening and closing valves in pipes and circuit breakers on the power grid, monitoring temperatures and pressures in reactors, and managing assembly line machinery. And because these systems are now connected to corporate networks, their vulnerabilities serve as an entree into the guts of a nation's critical infrastructure. A malicious hacker or terrorist group could conceivably take down parts of the power grid, throwing a country into darkness; they could take out emergency telephone systems or disable the floodgates to a dam. Even scarier to terrorism experts is a digital intrusion combined with a physical attack - think 9/11, but magnify the chaos by adding an electronic knockout of regional or national communication and power systems. The intent is clearly present: Raids in Afghanistan in early 2002 discovered that al-Qaeda operatives had scoured Web sites containing information on SCADA (supervisory control and data acquisition) networks in US water systems and the electricity grid.

Alarmism?

Unfortunately, the people with detailed knowledge of control systems security say no. Control systems are designed for efficiency and reliability - not security. In fact, "It requires very little knowledge" to hack into a control system, says Juan Torres, program manager of the SCADA program at Sandia National Laboratories.

Experts worry that this issue is not getting enough attention from both government and the private sector, for a variety of reasons: technical ignorance, lack of funding and perhaps the absence of a major incident to date. Even with a concerted public-private effort, securing these systems will take years. Older, legacy controllers can't handle newer security technologies such as encryption; in fact, many don't even have enough horsepower to accept operating system updates or software patches. "How a control system works is different from an IT system, technologically," says Joe Weiss, the former technical manager of the Electric Power Research Institute's Enterprise Infrastructure Security program, now an executive consultant with Kema. "It's deterministic, cheap and old, with little in the way of computing resources. It's not in any way, shape or form designed to be a secure system." Compounding these technical challenges are a number of entrenched cultural and management obstacles. The people generally responsible for managing control systems are engineers who often have had little cybersecurity training - or interest.

That's a lot of problems. And a recipe for potential disaster.

Efficient, but Not Secure

For years, distributed control systems and SCADA systems (see "Talk to Your Plants", right, for the difference) were designed with proprietary technology, and were physically and technologically isolated from the corporate networks that run standard IT applications.

Fatefully, the drive for efficiencies of cost and time led many companies to knock down the wall that traditionally separated those two types of networks. In the electric power industry in the US, for example, deregulation led to more interconnectedness as executives sought more information from control systems to help make output and pricing decisions. Manufacturing executives wanted to pull up real-time information from, say, their assembly lines, to monitor how efficiently their factories were running. "As the networking evolution came through and local and wide area networks were installed, they were generally installed by IT. Operations, so as not to spend double the money, started using the corporate LANs and WANs for the control networks," Weiss says. Ultimately, this meant many control systems were connected to the Internet.

This linkage has profound security implications. Now control systems are exposed - via the Internet, intranets, remote dial-up and wireless capabilities - to hacks, worms, viruses and other dangerous payloads. That exposure scares Jonathan Pollet, president of PlantData Technologies, who advises companies on control system security. "With each release of worms and viruses, there are more and more customers with downtime," he says. Pollet says the Sasser worm in spring 2004 took out several oil platforms in the Gulf of Mexico for two days. "They had firewalls, but worms crawled through commonly used ports like ports 80 and 139. If any type of connectivity is not turned off, a worm in a corporate network will crawl to control systems," he says. Another virus, SoBig, affected the dispatching and signalling systems of CSX Transportation, halting train service for four to six hours along the Northeast Corridor in August 2003.

Accentuating the connectivity problem is the growing move away from proprietary software toward standardized and off-the-shelf software and hardware. For example, Pollet notes that some SCADA software vendors use the same Microsoft connectivity tools found in products such as SQL Server and Exchange. "A worm written to take down a SQL server can take down a SCADA system that has nothing to do with the target server," says Pollet. The same vulnerabilities exist with other common technologies, from Unix to ActiveX.

Worlds Apart

Glance at the organizational chart of a typical large company and you'll see that cybersecurity falls under the purview of the CIO or, sometimes, the CISO. That makes sense; those execs are best qualified for the critical job of maintaining safe, secure and private IT networks. But who looks after the security of control systems? In most cases, Weiss says, the real answer is no one. The CISO knows IT security but nothing about the shop floor or the control systems. The VP of operations or manufacturing understands engineering and control systems but knows nothing about - and has no budget allotted for - cybersecurity.

John Maguire, senior security analyst at PJM, the world's largest electric grid operator (it covers a region that includes Baltimore, Chicago, Philadelphia, Pittsburgh and Washington, DC), sees firsthand the lack of operational security know-how. PJM's members include some 800 power sources, and Maguire serves as PJM's external security rep to those companies. He says security is a tough issue for PJM's membership. "We've pointed at the right documents and suggested best practices, but there's a fear of getting started or not knowing how to get started. It's a new set of responsibilities, and security isn't their core business. In [industries such as] banking and insurance, most of their business is about information. For our members, it's about producing electricity," says Maguire.

In fact, IT and operations groups are not just separate, they're often antagonistic toward each other, according to Weiss. The engineers responsible for control systems care about round-the-clock reliability. For that reason, all the workers with responsibility for, say, an electric power substation, might have the same user name and password to ensure no one forgets theirs if they're called into action at 2:00am to troubleshoot a system. If a CSO told them, wait, that's bad security, we need two-level authentication for anyone to gain access, it would just reinforce the perception among the field guys that IT doesn't "get" control systems, Weiss says.

Hampering a get-together between the two sides is the lack of an overall security policy in many companies. Torres says he is heartened by the fact that an awareness of the importance of cybersecurity policies is on the upswing, with urging from groups such as the North American Electric Reliability Council (NERC) in the power industry, for example. Torres believes more companies are putting such policies in place; however, he adds, "Whether they include the right things or not is another question."

Can't Patch This

In a typical corporate IT network, hundreds (or thousands) of PCs, servers and other devices are packed to the gills with processing power and memory. CSOs can slap on the latest security technologies without much adverse effect on the network. On the other hand, many legacy control systems still run on Intel 8088, 286 and 386 processors. These processors are "adequate for the functions they have, but if you try to lay, say, encryption over them, they can't handle it. We're sitting with 30- to 40-year-old systems," says Ken Watts, director of infrastructure and defence systems at the Idaho National Engineering and Environmental Laboratory, which does process control research for the Department of Energy.

The insecurity of these systems is manifested in plenty of other ways. Control system communications used to be proprietary; that changed when those systems began getting hooked up to enterprise networks and the Web. "The SCADA commands are now going over TCP/IP and clear text, and are highly vulnerable," says Pollet. "There's no way for a [control device] to know the SCADA command is what it says it is; there's no authentication, no encryption. They're highly vulnerable to denial-of-service [attacks] and viruses." He adds that easily downloadable TCP/IP packet-sniffing tools - such as Ethereal and Ettercap - can be used to read clear text. That would allow a hacker to read and capture user names, passwords and even commands.

SCADA systems also connect to a wide variety of other communications media - including public telecomms networks, wireless radio, and private microwave and fibre networks. In testimony before a House subcommittee looking into control system vulnerabilities in March, Gerald Freese, director of information security at American Electric Power, talked about the interdependencies between SCADA networks and the telecommunications functions that support them. "We have to keep in mind that telecommunications is vulnerable in its role as a transport medium. It is subject to attacks such as 'man in the middle', where transmissions are intercepted and altered, redirected or destroyed. Also, many power plants and substations use modems [vulnerable to a number of intrusion exploits] to manage equipment such as breakers, relays and switches over telephone lines."

Because SCADA systems were designed for efficiency and ease of use, vendors enable their products to be accessed remotely - through dial-up modems, wireless handhelds and the like - so that customers will have an easier time making fixes to systems, often with no authentication required. And companies often fail to install the same security measures on control systems - such as firewalls and intrusion detection systems - that they use to protect IT systems. But those technologies have their limitations as well, since they weren't designed with control systems in mind. For example, Weiss says a typical firewall filters Internet protocols such as TCP/IP but not control system protocols.

Patch management is another gnarly issue. The message from vendors sometimes seems to be: Patch at your own peril. That's because installing patches can interrupt the real-time functioning of the operating system, which could have bad consequences. "We had a control system supplier send out a warning letter to all its clients saying: Whatever you do, don't put in a patch for the Slammer worm. The patch will get you," says Weiss. Gary Sevounts, director of industry solutions at Symantec, notes that part of the problem is that it's difficult to test patches (or any other security technology) in an actual control system environment because of the requirement for 100 percent availability and predictable performance. "If there's a 10 to 15 percent hit on performance in a banking application, perhaps there's a delay, but the customer is probably OK," but the same 10 percent to 15 percent delay in a SCADA system can lead to a power blackout, Sevounts says.

Pollet points out another issue; vendors sometimes approve patches for only certain versions of software. He gives the example of a company that upgraded its operating system. "If I say my system isn't functioning, [control system vendors] ask what patch you're running. I say I'm running a patch for Windows 2003 Server. They say I can't give you any support [because that's not the OS our software works with]. They say scale back to the original OS. Companies can void a warranty by upgrading," says Pollet.

The Fix Is in - Sort Of

All these vulnerabilities raise the question: Are the major control system providers - including ABB, Emerson, GE, Honeywell, Invensys and Siemens - building more secure systems? Up until now, Weiss says those companies have focused entirely on improved performance, because that's what the buyers have asked for. Vendors responded by incorporating off-the-shelf software and hardware, and building Web and wireless connectivity into their products. But vendors are to blame as well. Instead of waiting for market pressures to force them into building more secure systems, they could take a more proactive stance and begin making a concerted effort to beef up the security of their products, and work more closely with customers to identify and mitigate the vulnerabilities of existing systems.

There are some examples of new efforts by vendors. Areva, a control system vendor, recently announced a new partnership with Symantec to strengthen the security of its products. Last year, software company Verano announced Industrial Defender, a product suite aiming to protect control systems from cyberattacks.

Meanwhile, the companies that use control systems aren't completely reduced to waiting for vendors to get their acts together. Pollet says better information security on the corporate network can greatly reduce the risks posed to control systems; he mentions better router configuration, antivirus software, intrusion detection systems and more diligent patching. Torres adds the nontechnology parts of the security equation: better configuration management, better documentation of network architectures, better patch management and better contingency planning. Above all, Torres thinks the cultural gap between the IT and control side needs to be bridged.

Various private industry and government groups are taking steps to make critical infrastructure companies more aware of the flaws in their control systems. The National Institute of Standards and Technology and the National Security Agency established the Process Controls Security Requirements Forum (members include reps from the electric, water, chemical and oil industries, as well as government labs and control system vendors) to develop security specs for control systems. NERC and the oil pipeline industry are working on the creation of permanent standards. Other government agencies and major critical infrastructure industries have established working groups to address the issue. Notably, last December, the US Department of Homeland Security created a new Control Systems Section inside the Protective Security Division of the Information Analysis and Infrastructure Protection Directorate.

But most managers, engineers and workers with day-in and day-out responsibilities for maintaining control systems may be a long way from putting cybersecurity on the front burner. Earlier this year, Weiss held a conference session attended by 30 to 40 people, some 15 of whom were plant managers. Weiss says that in his informal discussions afterward, every one of those managers thought cybersecurity had to do solely with the vulnerability of their e-mail systems. "They had no idea whatsoever about security around control systems," he says. Weiss observes that 9/11 served to make security a big deal in terms of physical and IT security: business systems, Web sites and the like. But control system security? "To this day, most people don't think they're vulnerable," he says.

Talk to Your Plants

Industrial control system networks generally fall into one of two types

Distributed Control System (DCS): These systems are used within a small geographic area, such as a manufacturing plant or nuclear reactor. A single vendor generally supplies the whole system: hardware, software, master control station, engineering workstation, programmable logic controllers (PLC), cabling and so on. A DCS is often connected over a LAN and may control the whole industrial process in a plant.

Supervisory Control and Data Acquisition (SCADA): These systems are typically used throughout a wider geographic area to distribute products such as electricity or oil. SCADA systems have a master that communicates with remote devices - PLCs or remote terminal units (RTUs), for example - over a cable, Internet, wireless, or a public or private switching network. An electric grid might have hundreds (or thousands) of remote devices that control the distribution of electricity. The RTU collects real-time data and sends it to a master station, which then sends commands back to the RTU to perform its functions.

Testing, Testing

Under the auspices of the Department of Energy's Office of Energy Assurance, US laboratories study control system vulnerabilities.

Tucked away in southeastern Idaho lies the Idaho National Engineering and Environmental Laboratory (INEEL), a 2300-square-kilometre site that has operated for more than five decades. The site's infrastructure (which includes nuclear reactors, chemical processing facilities, a 50-megawatt power grid and other key critical infrastructure elements) allows for the testing of - among other things - SCADA systems and security technologies in a real, integrated environment.

INEEL, in partnership with Sandia National Laboratories, is currently developing a national SCADA testbed. The plan, says Ken Watts, director of infrastructure and defence systems, is that vendors will agree to have their SCADA and process control systems tested for vulnerabilities. After that, it will be the vendor's responsibility to bring that knowledge back to the industry. (The eight-year, $US114 million program has hit some bumps recently due to funding constraints.)

Sandia, which began its SCADA research program in 1998, has also developed two educational courses around the design of secure SCADA systems and assessing the vulnerabilities of systems, and has offered those courses to utility representatives. Juan Torres, program manager of the SCADA program, says Sandia is working with the US Department of Homeland Security and others to develop a national strategy for process control security.

Join the newsletter!

Error: Please check your email address.

More about CSXDCSElectric Power Research InstituteHISHoneywellIntelInvensys GroupMicrosoftNational Security AgencySiemensSoftware WorksSymantecTransportationUS LaboratoriesVIA

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Todd Datz

Latest Videos

More videos

Blog Posts

Market Place