June 30, 2003
(Computerworld)
When an employee from an Australian company that makes manufacturing software got fired in early 2000, he applied for a job with the local government, but was turned down. In retaliation, he got a radio transmitter, went to a nearby hotel where there was a sewage valve, and used the radio to hack into the local government's computerized waste management system.
Using software from his former employer, he released millions of gallons of raw sewage near the hotel grounds and into rivers and parks.
"He did this 46 times before he was caught," notes Joe Weiss, a process-control cybersecurity expert and consultant at the Cupertino, Calif., office of Kema Consulting. "The first 20 [times], they didn't even know it was cyber," meaning an external attack launched using a computer, he says. "From 20 to 45, they finally figured it was cyber, but they didn't catch him until 46." Though this person never worked for the wastewater utility, he was still able to break into its supervisory control and data acquisition system, which was designed with a big security assumption in mind -- that only insiders would want to access it.
Hundreds of thousands of similar process systems and networks used in dozens of industries worldwide remain dangerously vulnerable. And like it or not, IT managers need to address this problem despite three enormous challenges: the traditional barriers between IT and the engineers who typically run process networks, the highly customized nature of process applications, and the lack of security software for process applications and networks.
Historically, IT has had little, if anything, to do with process-control systems, because they run reliably and rarely crash. Instead, IT focused strictly on corporate data networks. But that needs to change, experts say.
Process-control networks are to manufacturing environments what IT is to an office -- endemic. For example, more than 2,400 oil, natural gas and chemical companies in the U.S. employ process-control networks in their manufacturing systems. Other heavy users of process networks include the power, water, food, drug, automobile, metal, mining and manufacturing industries.
For example, process networks in the chemical industry control chemical-making equipment and monitor sensors. If anything goes wrong, such networks react by adjusting the environment in predefined ways, such as shutting off gas flow to prevent leaks or explosions.
One company that's taking process network security seriously and involving IT is Du Pont Co. in Wilmington, Del. Tom Good, a project engineer at the chemical manufacturer, has been leading its 20-month-old effort to categorize and reduce its process-control system vulnerabilities.
Du Pont's philosophy for dealing with this problem, he says, is that "on all of our critical manufacturing processes, we are either going to totally isolate our process systems from our business systems by not connecting our networks, or we're going to put in firewalls to control access."
To tackle process-control network security, Good says Du Pont formed a team made up of IT staffers, who understand networks and cybersecurity; process-control engineers, who understand the process-control equipment; and manufacturing employees, who understand manufacturing risks and vulnerabilities.
To give the three groups visibility, each reports to a separate member of a committee that's leading the effort. The team first discerned which control devices are critical to manufacturing, safety and continuity of production. Then the team identified the assets of each -- hardware, data, software applications -- and researched relevant vulnerabilities. Only then did it begin the arduous task of testing fixes and work-arounds to see which ones might work for which machines.
Even in a manufacturing environment that uses similar process-control hardware and software, precise vulnerabilities differ by environment. "Dealing with, say, a water treatment process on effluent out of a plant is considerably different than dealing with a production operation, where you might be dealing with vessels under high-temperature and high-pressure conditions," says Good.
On the basis of its research, the team is also deciding how to separate networks and where process-control firewall appliances should go. High-end enterprise firewalls aren't required; each process network supports only 10 to 50 users. "The greater cost is in the network equipment and re-engineering activities to separate networks and place critical process-control devices together on the clean side of the firewall," says Good. "The challenge is to accomplish these tasks on online control systems while keeping the process running."
The Challenge
Until about 15 years ago, most process-control networks were secure because they were extremely proprietary. Then customers demanded less expensive front ends and TCP/IP networking. Such networking opened up process-control systems to common vulnerabilities, then the Internet, yet the underlying systems weren't strengthened to make up the security difference.
The methods IT uses to secure data networks can't be used to quickly secure control systems. Every component can have multiple critical functions. For example, at a refinery, various sensors help ensure safety when filling tankers - whether there's a truck beneath the filler, whether there's gas available. If the truck's tanker reads full or the truck moves away, the process must interrupt.
Process-control hardware is also difficult to secure. Application customization is so rife that it's impossible to run antivirus software on some PCs and Unix boxes, says Kris Zupan, CEO and chief technology officer at e-DMZ Security LLC, also in Wilmington.
Likewise, patching the operating system can require rewriting the application. In the rare cases when applications can be patched, shutting down the always-on machines is costly -- and a patch might have unintended effects and compromise production.
Other information security tools don't work well in these environments either; for example, complicated passwords can slow access in an emergency. As a result, every machine of a particular type worldwide may have the same password. In other words, anyone who has ever used one type of machine theoretically has access to all machines of that type.
IT managers everywhere will have to learn that safety in process control is paramount. That's a switch. IT's traditional goal is "data confidentiality, data integrity and data availability," says Theresa Grant, director of information security at The Dow Chemical Co. in Midland, Mich. "However, in the process-control arena, the very first objective is safety, the second is safety, and the third is operational integrity." If certain systems fail, people can die.
To help companies assess and remediate process-control vulnerabilities, various initiatives are under way. The Instrumentation, Systems and Automation Society, a Research Triangle Park, N.C.-based standards body, is developing best practices for securing process-control networks.
Various organizations, including petroleum and chemical trade associations, have commissioned studies to find best security practices for their industries. Studies in hand, they hope to persuade each industry to implement security best practices.
Another initiative, at least at Du Pont, is to get Microsoft Corp. to better detail its patch practices. Good says too much time is spent testing patches to make sure they work with the process-control applications, when in many cases, the patch might not even be applicable.
Du Pont sat down with Microsoft representatives and "exposed them to our world of safety -- where any safety incident is unacceptable," says Good. "I don't know where the awareness-building will lead. If Microsoft measures what it's doing as 'better' because it had 200 fixes this year vs. 250 fixes last year, there's obviously a gap in meeting the level of performance that is important to our industry."
The more realistic solution is for IT and engineering departments to cooperate on the problem. For example, IT knows how to better secure things -- "change management, release management, providing things of that nature," says Dow's Grant. Engineers understand process-control intricacies.
Hence, any fix-it team needs both kinds of experts, as well as anyone else with a stake -- manufacturing, supply chain -- to help explain what touches what. That team would map computers, processes and networks and test where security can be applied. It's exacting work, warns Zupan: "If you modify the controlling system, it can produce hazards that not even the designers predicted."
The irony, of course, is that companies demanded off-the-shelf components, and now they're paying the security price. Network separation is likewise no simple panacea. "It's been there in limited fashion [before], but a lot of times, there's a lot of back doors and a lot of Web front ends that need to be protected as well," says Michael Rasmussen, an analyst at Forrester Research Inc.
Process-control hardware can have a life span of 15 years. No doubt, some security vulnerabilities will remain as long as current hardware remains in use. By tackling the greatest risks, however, companies can help minimize their process-control system vulnerabilities and better secure manufacturing environments.
Schwartz is a freelance writer in Somerville, Mass. He can be reached at Mat@Penandcamera.com.