Software Failure Cited in Blackout Investigation

Computerworld - The task force investigating the cause of the August blackout that crippled much of the Northeast and parts of Canada concluded that a software failure at Akron, Ohio-based FirstEnergy Corp. "may have contributed significantly" to the outage.
The interim report of the U.S.-Canada Power System Outage Task Force, released last week, highlights the failure of various IT systems that thwarted utility workers' ability to contain the blackout. It found no evidence that malicious insiders or external saboteurs were responsible for the outage.
According to the task force, FirstEnergy's Alarm and Event Processing Routine, a software program that gives operators visual and audible indications of events occurring on their portion of the grid, began to malfunction. As a result, "key personnel may not have been aware of the need to take preventive measures at critical times," the report states.
In addition, "some companies appear to have had only a limited understanding of the status of the electric systems outside their immediate control," the report concludes. "This may have been, in part, the result of a failure to use modern dynamic mapping and data sharing systems."
In a statement released Nov. 19, FirstEnergy President and Chief Operating Officer Anthony J. Alexander said the company remains "convinced" that the blackout can't be traced to any one utility system. "We recognize that our computer system experienced problems that day," he said. "After an extensive analysis, we submitted a report to the task force that identified a previously undetected flaw in vendor software that resulted in the loss of an alarm function, affecting our operators' understanding of events on our system."
However, "by focusing its analysis on a few selected events, the conclusions the task force reached don't address the complexity and magnitude of operations on the interconnected grid," he added.
Aside from the alarm software failure, the task force found that Internet links to Supervisory Control and Data Acquisition (SCADA) systems weren't properly secure. It also concluded that some operators lacked a system to view the status of electric systems outside their immediate control.
Of particular concern to the task force is the existence of direct and remote links between corporate networks used at utilities and the real-time SCADA systems used to manage the power grid. Until now, the electric industry has refused to publicly acknowledge these linkages and the vulnerability they pose. But the task force report puts SCADA system security at the center of the industry's most pressing security challenges.
"The existence of both internal and external links from SCADA systems toother systems introduced vulnerabilities," the report says. But it stops short of assigning blame for the blackout to a series of viruses and worms that hit the Internet prior to and during the outage.

Read more about disaster recovery in Computerworld's Disaster Recovery Knowledge Center.