IT leads recovery after regional power failure
Disaster recovery plans were put to the test; users report resilient systems
Computerworld - When the power went out in Manhattan late yesterday afternoon, the stock markets had already closed. But the crucial trade-settlement system that uses thousands of batch-processing computers around New York City to clear billions of dollars in trades had just come to life.
Diesel generators at brokerage, bank and clearinghouse data centers around Manhattan and New Jersey kicked in, and IT departments said that they were far better prepared for what most called a simple power outage than they were on Sept. 11, 2001.
The New York Stock Exchange said no data was lost from yesterday's trading as a result of the blackout. "In addition, the Securities Industry Automation Corp., which is our data processing and technology operations arm, is operating at normal capacity on generator power," a spokeswoman said last night.
Russ Lewis, CIO at GFI Group Inc., said the Wall Street-based online brokerage took a "hard hit around 4:12 p.m. ... and we went right into disaster recovery mode."
"All the systems did come down. We immediately went on generator backup for both our data center and our trading floor," Lewis said this morning. "Our systems all flipped over as well. Asia and London were unaffected because the systems flipped over properly."
As a precautionary measure, Lewis said, he performed end-of-week backups last night and sent them via the company's virtual private network to London, "in case we weren't able to get power into the New York office today and we had to shut the office down."
Lari Sue Taylor, director of enterprise information security and recovery at FleetBoston Financial Corp. in New York, said a 62-member crisis management team that was created after 9/11 began assessing the situation within an hour of the initial blackout.
FleetBoston, which has several offices in Manhattan, was forced to move workers to SunGard Data Systems Inc.'s facilities in Carlstadt, N.J. Taylor said the bank also had to transfer network operations for its Quick & Reilly online brokerage service to those facilities.
Diesel generators at Merrill Lynch & Co. in lower Manhattan revved up as the power went out, and computer systems in the Manhattan and New Jersey data centers didn't skip a beat, said spokeswoman Selena Morris. "We were obviously prepared if something like this happens," Morris said.
At Case Western Reserve University in Cleveland this morning, CIO Lev Gonick was running on two hours' sleep after having worked on recovering core systems, including e-mail, course management systems and enterprise systems, throughout the night.
Power was still out, and nearly 1,000 students were due to move into the university for the new school year tomorrow.
Gonick said school officials were "desperately concerned" about losing data on returning students' tuition payments and course information, but a storage-area network Gonick implemented after Sept. 11 took automatic snapshots of data sets as the power began flickering at 4:07 p.m. EDT on Aug. 14. He said today that he lost only a "fraction of a second" worth of data.
"When we got hit, we got hit with a double surge. It was on the second surge that some backplanes, and some network routers got hit pretty badly. We also think the second surge may have hurt some of our large servers as well," Gonick said. "We've got a couple of servers that are a bit cranky coming up. As soon as the system came up, we had to go back and match the last save. It's not been flawless. But it's been as close as I can imagine."
Similarly, Alan Winchester, a technology attorney at Harris Beach LLP in New York, said all of the law firm's financial records are replicated in real time to its Rochester, N.Y., office, which has a generator.
Winchester said disaster recovery lessons learned after Sept. 11 were quickly implemented at Harris Beach after the lights went out.
IT staff members left the building with backup tapes for Tuesday through today, he said. "We can always restore it if something crazy happens to the building," he noted. "We can also restore it if we need to get the information to a server in a part of the country that's not affected." The law firm has offices in several other locations, including Washington and California, as well as connections with other law firms that would help if needed, Winchester said.
FedEx Corp. said the lack of power at its hubs and stations in the blackout areas delayed the processing of package information because drivers couldn't download data from bar-code scanners into the FedEx network.
Bob Brewin, Linda Rosencrance and Todd R. Weiss contributed to this story.
Read more about Disaster Recovery in Computerworld's Disaster Recovery Topic Center.
- IT Security - Fighting the Silent Threat "IT Security - Fighting the Silent Threat" is a global report into business attitudes and opinions on IT security. Download the report now...
- Cutting Complexity - Simplifying Security This white paper looks at how the latest IT Systems Management solutions can simplify and automate a vast range of routine IT management...
- Your Data under Siege: Defeating the Enemy of Complexity Even if you have adequate antivirus protection, are there still holes in your IT security armor? Is lack of bandwidth to manage the...
- Build Your IT Security Business Case In this latest whitepaper from Kaspersky Lab, you'll find useful facts, examples and business case arguments to help you get buy-in and commitment...
- Pre-Engineered solutions from VCE Simplify Core Infrastructure Implementation In this video, the CTO of Purdue Pharma, a privately held pharmaceutical company explains how Purdue transformed their data center infrastructure with VCE.
- Data Protection and Disaster Recovery with iSCSI and VMware Get this on demand webcast now All Disaster Recovery White Papers | Webcasts