Surviving the big one: 7 lessons learned from the decade's deadliest disasters

Computerworld - You may have a disaster recovery plan in place, but are you prepared for a true catastrophe?

If you're fortunate, you don't have personal experience dealing with tragedies like the Sept. 11 terrorist attacks or Hurricane Katrina. But one of the best ways to prepare for such disasters is to learn from those who survived them.

Tulane University, Hard Rock Casino & Hotel, Hancock Bank and others learned some tough lessons during the decade's deadliest disasters. And now they're sharing their hard-won knowledge. Have you survived a disaster? Share your tale.

1. Plan a backup site in advance

THEN: After Hurricane Katrina hit, Tulane University's IT team was able to recover its backup tapes, but the New Orleans data center was without power, and no backup site had been prearranged.
NOW: Signed on as a SunGard customer, the university is entitled to a mobile data center that could be used for local processing. And backup tapes are now sent to Baton Rouge three times a week. Read more about Tulane's recovery efforts.

2. Take control of your telecommunications

THEN: Like many other companies, insurance firm F.A. Richard & Associates hadn't expected the local telecommunications failure triggered by Hurricane Katrina.
NOW: FARA signed up with various cell phone providers using multiple area codes. The firm can also now reroute its own 800-number services in emergencies, via the Web. Read about other ways FARA took control.

3. Be ready for extended power outages

THEN: When Hurricane Katrina devastated the Louisiana coast, a lack of batteries and power had Hard Rock Hotel & Casino employees relying solely on text messaging to communicate.
NOW: As part of its revamped disaster recovery plan, IT employees carry car chargers for their cell phones. There's a new personnel plan for disasters, too. Read more about Hard Rock's disaster communications plans.

4. Choose a "low-threat" data center location

THEN: During Hurricane Katrina, Hancock Bank's Gulfport, Miss., data center, just one half mile from the Gulf of Mexico, was devastated.
NOW: The bank's new $16 million data center is still in Gulfport, but the hardened, lights-out facility is located farther inland on the highest point in the area. It can withstand 200 mph winds. Read more about Hancock Bank's rugged data center.

5. Speed up server file replication

THEN: When Katrina struck, Hancock Bank's virtual server files could be quickly set up on hardware in a backup data center. But getting them there and loading from tape took 36 hours.
NOW: A new system reduces the boot recovery process to about 45 minutes. Read more about Hancock Bank's new DR plan.

6. Layer on communication methods

THEN: As communication channels began to flicker back on in Katrina's aftermath, companies throughout the region, including Marriott, found that different telecommunications components were fading in and out.
NOW: Marriott's recovery teams learned that the best way to keep communication channels open among employees was to use a mix of cell phones and BlackBerry devices with different carriers. Having discovered that pin-to-pin communication and texting were the most reliable solutions, the disaster preparedness team quickly educated employees on the features and made it part of the disaster recovery plan.Read more about Marriott's IT disaster recovery teams.

7. Create a mirrored infrastructure

THEN: In 2004, when Hurricane Gaston stalled over Richmond, Va., for hours, Estes Express Lines' first-floor data center was awash in four feet of water. Company executives watched helplessly as 185 terminals used to direct the operations of more than 20,000 tractor trailers just died. All told, Estes had $16 million in hardware losses.
NOW: The company pieced together a new infrastructure, complete with software that allows data to be whisked off-site immediately. That new architecture is mirrored in a hurricaneproof backup site nestled in sunny and dry Mesa, Ariz. Read more about Estes' battle with Gaston.

NEXT: New strategies needed for new disasters

Read more about storage in Computerworld's Storage Knowledge Center.