As last night's blackout slowly receded, power company CIOs and utilities experts said similar or more catastrophic failures are possible if the industry and government fail to develop more modern control systems.
"It's not just bigger servers or better databases that we need," said Ali Jamshidi, vice president and CIO of First Energy Corp. in Akron, Ohio. "We just don't have the analytical tools that can do analysis on a real-time basis and that are predictive vs. reactive. Frankly, the tools that are available are just not robust enough."
Out of First Energy's 4.3 million customers, 1.5 million were affected by the blackout.
Jamshidi added that the stability of the grid is unlikely to improve unless industry and the government invest more time and money in developing advanced software that can serve as a real-time decision-support system for electric grid managers and operators.
"There needs to be a more concentrated and cooperative approach at the federal level. Otherwise, these kinds of failures will continue to be difficult to predict," he said.
Mark Ascolese, president of Powerware Corp., a Raleigh, N.C.-based firm that manufactures power management software for the energy industry, agreed that lack of investment in the right technologies has contributed to the U.S. power grid's poor state of health.
"What's not been invested in during the last 40 years is the infrastructure for transmission and distribution, including the hardware and software that power SCADA systems," he said, referring to Supervisory Control and Data Acquisition Systems, which are real-time computers used to manage grid capacity.
Joe Weiss, an analyst at Kema Consulting in Fairfax, Va., and former technical manager of the Enterprise Infrastructure Security Program at the Electric Power Research Institute in Palo Alto, Calif., said the situation caused by the badly outdated technologies is compounded by the highly interconnected nature of the grid, which makes such widespread cascading failures an ever-present possibility.
Weiss also acknowledged that much of the research and development work in more resilient IT systems for the electric power grid -- such as the "intelligent grid" initiative called for last year by the National Research Council -- haven't made their way into operation to the extent officials would like.
Howard Schmidt, chief security officer at eBay Inc. and former chairman of the President's Critical Infrastructure Protection Board, championed the R&D effort in security systems capable of operating in the real-time environment of the electric grid. He said IT systems capable of providing an adequate amount of security and reliability for the nation's power grid don't yet exist and that their development is one of the most pressing issues facing the homeland security and R&D communities.
"There's better security at some e-commerce sites than there is on some of our electric grid systems," said Schmidt.
And IT security has taken on new meaning for the energy industry in light of last week's failure, said Schmidt and other industry experts.
Jamshidi agreed with Schmidt's assessment, calling the blackout the most realistic security drill possible, one that exposed serious weaknesses in the system, including the threat from deliberate physical and cyberattacks. "This could have been even more disastrous," said Jamshidi. "Clearly, a well-informed attacker with information on the strengths and weaknesses of the grid could cause a much more damaging outage."