IT leads recovery after regional power failure
Disaster recovery plans were put to the test; users report resilient systems
Computerworld - When the power went out in Manhattan late yesterday afternoon, the stock markets had already closed. But the crucial trade-settlement system that uses thousands of batch-processing computers around New York City to clear billions of dollars in trades had just come to life.
Diesel generators at brokerage, bank and clearinghouse data centers around Manhattan and New Jersey kicked in, and IT departments said that they were far better prepared for what most called a simple power outage than they were on Sept. 11, 2001.
The New York Stock Exchange said no data was lost from yesterday's trading as a result of the blackout. "In addition, the Securities Industry Automation Corp., which is our data processing and technology operations arm, is operating at normal capacity on generator power," a spokeswoman said last night.
Russ Lewis, CIO at GFI Group Inc., said the Wall Street-based online brokerage took a "hard hit around 4:12 p.m. ... and we went right into disaster recovery mode."
"All the systems did come down. We immediately went on generator backup for both our data center and our trading floor," Lewis said this morning. "Our systems all flipped over as well. Asia and London were unaffected because the systems flipped over properly."
As a precautionary measure, Lewis said, he performed end-of-week backups last night and sent them via the company's virtual private network to London, "in case we weren't able to get power into the New York office today and we had to shut the office down."
Lari Sue Taylor, director of enterprise information security and recovery at FleetBoston Financial Corp. in New York, said a 62-member crisis management team that was created after 9/11 began assessing the situation within an hour of the initial blackout.
FleetBoston, which has several offices in Manhattan, was forced to move workers to SunGard Data Systems Inc.'s facilities in Carlstadt, N.J. Taylor said the bank also had to transfer network operations for its Quick & Reilly online brokerage service to those facilities.
Diesel generators at Merrill Lynch & Co. in lower Manhattan revved up as the power went out, and computer systems in the Manhattan and New Jersey data centers didn't skip a beat, said spokeswoman Selena Morris. "We were obviously prepared if something like this happens," Morris said.
At Case Western Reserve University in Cleveland this morning, CIO Lev Gonick was running on two hours' sleep after having worked on recovering core systems, including e-mail, course management systems and enterprise systems, throughout the night.
Power was still out, and nearly 1,000 students were due to move into the university for the new school year tomorrow.
Gonick said school officials were "desperately concerned" about losing data on returning students' tuition payments and course information, but a storage-area network Gonick implemented after Sept. 11 took automatic snapshots of data sets as the power began flickering at 4:07 p.m. EDT on Aug. 14. He said today that he lost only a "fraction of a second" worth of data.
"When we got hit, we got hit with a double surge. It was on the second surge that some backplanes, and some network routers got hit pretty badly. We also think the second surge may have hurt some of our large servers as well," Gonick said. "We've got a couple of servers that are a bit cranky coming up. As soon as the system came up, we had to go back and match the last save. It's not been flawless. But it's been as close as I can imagine."
Similarly, Alan Winchester, a technology attorney at Harris Beach LLP in New York, said all of the law firm's financial records are replicated in real time to its Rochester, N.Y., office, which has a generator.
Winchester said disaster recovery lessons learned after Sept. 11 were quickly implemented at Harris Beach after the lights went out.
IT staff members left the building with backup tapes for Tuesday through today, he said. "We can always restore it if something crazy happens to the building," he noted. "We can also restore it if we need to get the information to a server in a part of the country that's not affected." The law firm has offices in several other locations, including Washington and California, as well as connections with other law firms that would help if needed, Winchester said.
FedEx Corp. said the lack of power at its hubs and stations in the blackout areas delayed the processing of package information because drivers couldn't download data from bar-code scanners into the FedEx network.
Bob Brewin, Linda Rosencrance and Todd R. Weiss contributed to this story.
Read more about Disaster Recovery in Computerworld's Disaster Recovery Topic Center.
- Server-side Caching for the VMware Admin vExpert David Davis weights in on how best-in-class server-side caching solutions can drastically improve storage performance and reduce latency without the addition of...
- Top 5 Reasons for Cloud-Based Disaster Recovery There is no question that every business wants to protect their operations from downtime and loss of data. But many companies don't have...
- 5 Things You Didn't Know About Cloud Backup IT departments are embracing cloud backup, but there's a lot you need to know before choosing a service provider. Learn all the critical...
- Case Study: Extending DR Protection for Apps W/O Fixed Costs/Fees Find out how the city of Asheville, NC won the Global City on a Cloud Grand Prize from Amazon AWS for Best Practices...
- Is SQL Server AlwaysOn really as powerful? Tips and Tricks from the field With the introduction of AlwaysOn, Windows Clustering Services is now more critical than ever.
- Introducing Cloud-Based Disaster Recovery From VMware Cost-effectively protect your business applications in the case of a local disaster or disruptive event. VMware is excited to introduce vCloud Hybrid Service... All Disaster Recovery White Papers | Webcasts