September 16, 2003
(Computerworld)
DALLAS -- East Coast data center managers at an IT conference here were keeping an eye on Hurricane Isabel, calling their offices to ensure that backup-generator fuel tanks were topped off. But some data centers may be better prepared than others in dealing with the storm, thanks to last month's massive regional power failure.
The Aug. 14 blackout, which left some companies without power for several days, is having a lasting impact on some data center managers. Companies hit by the outagesay they are more likely to increase future data center training and test power supplies and recovery plans than those that were unaffected by the outage, according to a survey of 500 IT managers by Orange, Calif.-based data center user group AFCOM.
Indeed, the survey details a sharp contrast in how those affected last month will respond to potential disasters compared with those that weren't affected. Companies that lost power say they will spend more time, for instance, on backup power testing and staff training, while many unaffected businesses will focus on vendor-developed contingencies, such as backup hot sites.
"What you're seeing there is a little bit of pain," said Don Tissell, a server facilities manager at Frito-Lay Inc., referring to affected data center managers' emphasis on fixing their own problems instead of calling on vendors for solutions. His Plano, Texas-based company suffered only a limited impact from the blackout, which it handled with backup power. "Living it and reading about it are two different things," he said.
That's true for Juan Sierra, computer operations manager at Visiting Nurse Service of New York. "Whoever went through the power outage is definitely rethinking what you have. You have to," said Sierra. His organization lost power during the blackout, and he is now working on improving training and backup power supplies.
According to AFCOM's survey of data center managers, 19% of those affected said they have plans to test backup power devices, compared with 8% of those who were unaffected; 18% of those who lost power in the blackout will be conducting additional training, as opposed to 8% of those unaffected by the blackout. And 17% of the affected data center executives plan to test existing disaster recovery plans, while just 9% of those who were unaffected will be doing so.
Among unaffected companies, however, 24% plan to implement a hot site to manage backups, compared with just 4% of the affected companies. Vendors typically provide such backup site services.
The message from affected companies is "be prepared," said Donovan Oliver, the production manager for computer operations at Girl Scouts of the USA in New York. "You never know when stuff like this is going to happen."
The survey also measured the cost of the outage for data center operations. Two percent of those surveyed reported that they lost more than $10 million because of the blackout; 1% reported losses of between $1 million and $5 million. And 10% reported losses of $100,000 to $500,000.
More than half of those affected by the blackout, however, said that associated costs were less than $10,000 -- a figure some were skeptical of. Among those who doubted the accuracy of those low cost estimates was Dennis Reid, operations manager and the person in charge of contingency planning for Time Customer Service Inc., the order fulfillment center for publishing company Time Inc.
Reid said those companies that reported little financial impact either had recoverable services or aren't really sure what the outage cost them. He suspects that the latter is true for many businesses. Unless a company has gone through a due diligence process on disaster recovery, it probably doesn't have a good handle on the cost of the outage, Reid said.