What would you do if you found out some of the software running your company was behind schedule on dozens of patches and updates — including critical security patches that were several years out of date? You might think that could never happen. You'd be wrong.
CIOs in enterprises across many industries are discovering that surprisingly antiquated and purpose-built applications are supporting their companies' operations, controlling everything from factory equipment to pumps, heating and cooling systems and electrical power.
This is the world of operational technology, a world that has existed ever since the first engineer put a speed limiter on a steam engine in the 1800s. Since that time, operational technology has run completely independently from IT in most organizations. But these days, several converging forces — increasingly complex digital devices, growing concerns over security, and a desire to benefit from big data and the Internet of Things — are forcing operations and IT to work together. It's not always easy.
"Operational technology is the domain of engineering and operations staff that is separate from IT," says Gartner analyst Kristian Steenstrup. "In most companies, they're in separate buildings and never talk to each other. How do you manage things in your organization when you've got more technology and data traffic outside IT governance than within?"
This may sound like shadow IT, but it's very different, he warns. Shadow IT occurs when a department with a tradition of using technology provided by IT chooses to go outside that framework for some reason, usually frustration over IT's rules about which technology to use or impatience at how long IT takes to evaluate and adopt a new technology. It nearly always involves a relatively new, relatively small and often cloud-based deployment.
"With the wave of a hand, IT departments could look after shadow IT," Steenstrup says. "IT departments as they're currently structured could not run operational technology. They're unfamiliar with the architecture, vendors, applications, availability requirements and real-time criticality involved. It's completely different from what most IT departments do."
Beyond the 'air gap'
So why can't IT and operations keep on operating separately, as they've done for years? Because changes at most industrial organizations have rendered that approach impractical. First and foremost are growing concerns from the C-suite about cybersecurity, sometimes coupled with worries about compliance with newly enacted government regulations.
"The thing that's happened in the last three to five years that's really pushing these guys together is security," says Peter Zornio, chief strategic officer at Emerson Process Management, part of St. Louis-based Emerson Electric, a $24 billion global manufacturing company. Emerson Process Management provides automation equipment, solutions and services for businesses in the oil and gas, refining, and chemicals industries, among others. "Security concerns around running these facilities are very different from what IT is looking at," Zornio says. "IT is usually concerned about someone coming in and stealing information. On process control, you're worried about people coming in and changing something in the equipment running a process. Some of these are dangerous processes that, if disrupted, could result in accidents, injury, environmental damage or even death."
As C-suite executives grow ever more aware of these dangers, they're increasingly concerned that operations be secured, something that techies know how to do and engineers don't. In the past, many industrial enterprises kept their operational technology relatively safe with an "air gap" — meaning operations equipment was not connected to the Internet or the rest of the corporate network.
But that approach means sacrificing the efficiencies and just-in-time inventory savings that become available when a manufacturing company gives its suppliers and other business partners a direct connection to its systems. It also impedes the data analysis that can benefit the bottom line by providing the information necessary to, for example, direct material to machines that are operating below their capacity, predict what customers will want or identify machines that may soon need maintenance or replacement. As top executives start seeking some of those benefits — often by asking IT for them — the air gap security strategy looks less and less feasible.
Most operations engineers aren't trained to deal with security. That was the case at Hunter Douglas North America, part of $2.4 billion Hunter Douglas, a maker of window treatments and architectural products based in Rotterdam, Netherlands. This year, the IT department and product engineers collaborated on a new motorized window treatment that could be controlled by a smartphone app. The IT folks were surprised that the product engineers hadn't given much thought to cybersecurity, says Hunter Douglas North America CIO Rob Meilen.
But, he notes, if your job has always been to design window treatments, and you've never before connected one to the Web, it makes sense that cybersecurity might not be top of mind. "Our teams in IT weren't quite sure how to deal with a group of product engineers who were wondering why you'd design that in from the outset," he says. "They weren't resistant, but they had never thought about it. Thinking about cybersecurity is second nature to us."
Another factor pushing operations and IT to work together is the way operational technology is evolving. The software that runs operational devices is increasingly dependent on traditional IT operating systems, often Windows. That's a trend that's likely to continue because the economics of it make too much sense for it not to. "Companies like ours use off-the-shelf technology because it's a lot more cost-effective than creating it ourselves," Zornio says. "It makes integration with IT systems easier and takes advantage of the higher cost-to-performance ratio of a Dell PC. Dell's making millions of them; if we were doing it, we'd be making a couple of thousand."
This means that operations engineers, trained and accustomed to dealing with heavy equipment, now find themselves managing a growing portfolio of software, often Windows-based. They must also manage the frequent updates and patches that go with Windows — and that's a problem.
"Our systems were becoming more and more sophisticated, and more and more digital. Our people were looking more and more like IT specialists and less like engineers," says Chris Heimgartner, assistant general manager for distribution and engineering services at Snohomish County Public Utility (SnoPUD) District No. 1 in Everett, Wash. "We don't have the skill set," he says. "Engineers fix things when they're broken."
So, like a growing number of engineering chiefs, Heimgartner turned to SnoPUD CIO Benjamin Beberness and his IT team for assistance with operating system patching and maintenance. When Heimgartner sat down to discuss this change with one of the engineers who worked on supervisory control and data acquisition (SCADA) software, he asked whether SnoPUD's engineering team was good at handling patches.
"That's when he pulled out the patch record," Heimgartner recalls. It turned out one of SnoPUD's SCADA systems was behind on 28 rounds of patches, and some critical patches were three years out of date. "He said, 'No, we're not good at this.'"
'You're not the boss of me'
Though many IT and operations leaders recognize a growing need to work together, there are still some formidable obstacles. For openers, there can be deep distrust between IT staff engineering personnel (especially the latter), who are accustomed to operating independently.
"You have a little bit of 'You're not the boss of me' going on," says Rick Davidson, president of Cimphoni, a Delafield, Wis.-based consultancy that provides IT leadership services and IoT solutions across many industries.
Though he's now in a tech leadership role, Davidson spent years on the manufacturing side, so he has a good view of both disciplines. "When we in the factory look at IT, what we typically see is an insular organization," he says. "An engineer will say, 'All you want to do is layer on a lot of bureaucracy and controls that aren't needed.' The reality is, some of it is needed. But IT is working against the desires of a lot of guys on the factory floor who just want to do their own thing."
Even when engineering staffers welcome IT's assistance, IT experts may be reluctant to deal with the technology that runs operations. And you can't blame them. "Often, there are more problems than there are solutions," Steenstrup says. "Some are cultural and organizational. Engineering and IT in some companies have no track record of working together. And if IT gets involved, they're going to inherit seven different vendors they've never heard of, running outdated technology that can't be patched and often isn't architected in a way that makes it easy to manage."
For example, "There are a lot of operational technology systems running on Windows 95 and Windows NT 4.0," he says. "The vendors of these systems are also engineers, and they like stability so they haven't been as diligent about sending out patches or even approving updates," he explains.
Even when they have, sometimes their efforts to keep their software patched are rebuffed by their engineer customers. "We used to think we needed to issue major releases every year and a half, and our customers said, 'Whoa, too fast! Three years is great,'" Zornio reports. "The IT industry has somehow managed to program CIOs with the idea that of course you're always upgrading and we're not going to support this thing that's over six years old. And they think that's just the way it is. But engineers don't think that way, any more than you'd think you have to replace the engine in your car just because it's 10 years old."
Underlying these differing attitudes to patches is a fundamental difference in approaches, priorities and even work culture, born of the profound differences in the purpose and workings of operations and IT. You can begin to understand this difference by considering a simple action that serves as the first step in troubleshooting most IT problems: the reboot. Rebooting a PC, server or network is a routine matter to IT. But when a system runs a 24/7 manufacturing operation, a reboot that brings things to a halt isn't usually an option. Neither are planned downtime, updates and upgrades that are handled on weekends, or many other practices that are a normal part of managing corporate IT.
"The problem-solving approach is actually quite different," Steenstrup says. "If you have an engineering problem-solving requirement, you look at physical characteristics, temperature, weather and what might have been done in a similar situation elsewhere. You iterate it for your environment, get it running as well as it can, and then you lock it down or weld it in place."
Hunter Douglas North America's IT team encountered that mindset in product engineers as it was putting the finishing touches on the company's app-controlled window treatment. "Someone on the engineering side made the comment, 'We built the app. I think we're done,'" Meilen recalls. "The IT folks scratched their heads and said, 'You know . . . generally apps evolve over time.'"
That sort of evolution comes naturally to IT, especially in this era of lean and agile development. "IT says, 'Here are the technology standards. Here's the cheapest, soonest solution we can do, and then we'll prepare for change. We'll be upgrading later,'" Steenstrup says. "It's two completely different psychologies, neither of which can be discarded."
Get rid of the stereotypes
Understanding that both attitudes are legitimate and that the two must be combined is the first step toward effective IT-operations alignment. "Engineers are not really good at patching, and not as good as IT is at software deployment and making changes to technology," Beberness says. "But IT's not as good as they are at knowing what the impacts of a change would be."
Engineers complain that IT doesn't "get it" when it comes to operational technology, but Beberness says, "I think they do. Sometimes they need to be educated a little bit, but it doesn't take long for them to understand the criticality and urgency around operational technology." That being the case, he adds, "both organizations need to get rid of the stereotypes and get their teams working together."
Sounds good — but how exactly do you make that work? Begin with a frame of reference that will help engineers understand the bigger picture, Davidson suggests. "As an engineer, having a mental model — a picture from the shop floor through execution, through a traditional MRP [material requirements planning] system — is a big help," he says. "An engineer could look at that and say, 'Now I understand what my role is in the larger scheme of things.'"