One Year Later, IT Prepares for Next Disaster

Computerworld - Asiff Hirji, CIO at Ameritrade Holding Corp., said his business continuity plan for the company's Jersey City, N.J., data center worked flawlessly when the lights went out in much of the Northeast last August. Well, almost flawlessly. Ameritrade had 36 hours' worth of diesel fuel in its backup generators, and the main power grid wasn't fully restored for two days.
Ameritrade has since increased the amount of diesel fuel it keeps on hand, and Hirji has made other changes in response to the blackout. "We learned a bunch of things because in every plan, no matter how good, there is some little detail you forgot," he said last week as the one-year anniversary of the outage approached.
The blackout, which struck last Aug. 14 and left more than 50 million people without power for days, has given many IT managers food for thought over the past year. In interviews last week, a half-dozen IT executives said they have modified their business continuity plans in the wake of the blackout, investing in projects such as reinforcing their networks and increasing the amount of testing they do.
Drew Hiltz, deputy CIO at CDC Ixis North America Inc., said the New York-based asset management company experienced a communications meltdown for two days as a result of the blackout. In June, CDC Ixis completed a blackout-driven deployment of a Sonet ring that connects multiple offices in New York and New Jersey at OC-48 port speeds.
"As a result of the blackout, the central office had significant problems," Hiltz said. "Their equipment fried." The company is using the Sonet ring not only for data links but also for voice communications, which Hiltz said has helped increase communications resiliency. In addition, the company's IT staffers have worked with Verizon Communications Inc. to reposition circuits so they can support different offices around Manhattan.
Hiltz wouldn't say how much the Sonet ring cost to install. It's more expensive than the T3 lines that CDC Ixis had before the blackout but less costly based on the amount spent per megabit of bandwidth, according to Hiltz. And officials felt they had no choice. "Resilience is something we couldn't ignore anymore," he said.

Bill Moore, telecommunications manager at the Museum of Modern Art in New York, said all of the museum's IT and building operations systems went down during the blackout. At the time, the museum was undergoing a major renovation and expansion, so IT staffers asked Verizon to install two points of presence for redundancy. They also added the museum's