In retrospect, putting a data center on the fifth floor of a glass high-rise office building just a half-mile from the Gulf of Mexico wasn't such a good idea. During Hurricane Katrina, Hancock Bank's Gulfport, Miss., headquarters building was devastated. Today, the most visible change to the bank's disaster preparedness plan is a new $16 million data center farther inland, but that's just one of many changes Chief Operating Officer Shane Loper says will pay off when the next disaster strikes.
The bank now operates on a "4/24" plan that requires customer-facing systems to be operational within four hours of a disaster, and core business systems within 24 hours. "All of the things we are doing come with a price," Loper says. But because the bank is regional, it needs to ensure that a local disaster in Gulfport won't affect its other branches.
"You want to locate your data center in an area that has the lowest threat profile possible, and that means separating it from the headquarters offices and away from downtown areas when possible," says Stephanie Balaouras, an analyst at Forrester Research Inc.
Hancock's new data center is still in Gulfport, but the hardened, lights-out facility is farther inland on the area's highest point. It can withstand 200 mph winds, can be managed remotely and has dual 820-kilowatt generators with enough fuel to stay up and running 24 hours a day for a month.
The old data center's server infrastructure had been mostly consolidated using VMware virtual machine technology when Katrina struck. "Using more advanced techniques such as server virtualization to enable high availability and disaster recovery are good best practices," says Balaouras. The virtual server files and associated data were backed up and could be quickly set up on hardware in a backup data center in Chicago. But the challenge was getting them to Chicago and loading them from tape.
"[Just] the tape restoration process required 16 hours," and 36 hours elapsed before all systems were up and running, says Jeff Andrews, vice president and manager of information security.
The new system replicates virtual server files and data over an MPLS network, reducing the boot recovery process to about 45 minutes.
Disaster drills have also changed. "When I was driving up to Chicago [after Katrina], I was scared to death, because I knew there were things we never tested," says Andrews. Today, everything is tested under a full production load. Read about the do's and don'ts of disaster drills.