Hurricane, Floods Put IT Staffs to the Test

Disaster recovery works for some, but scope of calamity was difficult to prepare for

Katrina Coverage
At 2 a.m. on Aug. 27, two days before Hurricane Katrina devastated the Gulf Coast, Tim Babco grabbed a red binder containing the latest version of SCP Pool Corp.'s disaster recovery plan, put his dog and cat in the car, locked up his house and drove 500 miles from Covington, La., to the company's emergency operations center in Dallas.

Babco, senior director of IT at Covington-based SCP, a $1.3 billion wholesale distributor of swimming pool supplies, had relocated his operations on two earlier occasions when hurricanes threatened neighboring New Orleans. Both of those storms turned out to be near misses, but Babco said last week that the practice runs helped him fine-tune his plan for when the real thing finally hit.

"People would be lying to say these things always go perfectly," Babco said. "But has it succeeded in allowing our business operations to continue to buy, sell and distribute products? It certainly has, and that's what disaster recovery is all about."

However, the kind of disaster recovery planning done by Babco isn't universal -- especially for a calamity as massive as last week's. Gartner Inc. analyst Simon Mingay said that about 40% of Fortune 1,000 companies aren't prepared for a regional disaster. And small and midsize businesses are even less ready, he added.

"Obviously, we're looking at a level of devastation here that few would have considered," Mingay said. "But most still believe that these are things that don't happen to them."

Mingay said companies that have prepared properly for disasters, such as SCP, have extensive emergency communications plans, hot sites from which they can continue business operations for an extended period and some level of IT systems redundancy outside of their headquarters region.

But even companies that are well-prepared might not take into account a crisis of the magnitude as the one spawned by Katrina and the flooding that followed the storm.

For example, John Wade, CIO at Saint Luke's Health System Inc. in Kansas City, Mo., said his contract with a disaster recovery vendor allows the IT department to work out of a hot-site facility for up to six weeks. But in a disaster similar to Katrina, the six-week limit could "pose a hardship," Wade said.

Joe Hartman, an application development manager at HydroChem Industrial Services Inc. in the Houston suburb of Deer Park, said the company's disaster recovery plan involves moving corporate operations to facilities on the city's north side. "Anything that would completely wipe out Houston would leave us in a bad way," he added.

Babco said SCP's headquarters were unscathed by Katrina and the flooding that followed but lost all data and voice communications links. Fifteen of the company's 40 IT workers have relocated to Dallas for now, while another 15 were dispersed among offices around the country.

Three years ago, Babco decided to flip-flop SCP's primary and secondary data centers, placing its critical systems in the Dallas facility, which is run by Houston-based VeriCenter Inc. He also created an IT disaster recovery team for functions such as coordinating help desk services and relocating hardware. In addition, SCP has set up an internal business-continuity Web site that posts corporate alerts and provides toll-free numbers and an extensive list of employee contact information.

Not everything went completely smoothly last week. On Tuesday, Babco made an unplanned trip from Dallas to SCP's headquarters to retrieve 12 application servers, including ones supporting human resources and e-mail.

He also said he has been unable to get a response from Boston-based Iron Mountain Inc., which handles off-site data storage for SCP. The area around Iron Mountain's Kenner, La., facility is inaccessible, said Babco, who added that he would have liked to have had SCP's backup tapes sent to Dallas in advance of the storm.

"They've not been able to provide us any information about when they will be able to get tapes out of their facility," Babco said. "I think they weren't proactive in seeing the event looming and getting the tapes out of harm's way." He vowed to switch to a different storage archiving vendor.

Ken Rubin, senior vice president of marketing at Iron Mountain, said the company moved requested data tapes to customers' facilities beginning Aug. 26. It also put the tapes in its sites on higher shelves. "We don't know once the levees broke what impact or not there is to any of the contents of the buildings," Rubin said. "We're trying to get in."

Richard Kerley, chief operating officer at Shutts & Bowen LLP, said three of the Miami-based law firm's offices in Florida were hit by multiple hurricanes last year. "We could have had a situation where our main [data center] site in Miami was taken out, and the cold site in Orlando was taken out, and we were decapitated," Kerley said.

Six months ago, Shutts & Bowan signed up for data center space in a reinforced concrete bunker owned by Terremark Worldwide Inc. in Miami. Kerley said the law firm's neighbors in the bunker include the State Department, the Department of Defense and the Federal Emergency Management Agency.

Carol Sliwa contributed to this report.

Copyright © 2005 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon