The good news was that Tulane University's IT disaster plan specified how to prepare for a hurricane. The bad news was that it didn't say how to recover from one.
"We had a plan that said when there was an impending storm, how we'd shut things down, how we'd do the backups and how we'd protect the equipment -- and all that worked fine," says CIO Paul Barron, discussing how Tulane girded for Hurricane Katrina. "But we hadn't thought through what we'd do if the disaster actually occurred."
But Barron and his IT cohorts have thought about that a good deal since the deadliest and costliest hurricane in U.S. history slammed into New Orleans on Aug. 29, 2005. They have sharpened the preparedness part of the plan, laid out in detail how to recover from a storm and extended the plan to cover other kinds of disasters besides hurricanes.
After Katrina, backup tapes prepared before the storm were recovered intact, but the power was out in the data center and nobody knew where to send the tapes for processing. No backup site had been prearranged. Fortunately, SunGard Computer Services was able to offer some spare capacity in Philadelphia, even though Tulane did not have a disaster recovery contract with SunGard. After considerable debate, the tapes were sent there.
Getting payroll up and running again was the university president's top priority. "That would make a statement that we were still here and that the university was still functioning," says Barron. "Plus a lot of people needed the money." Tulane missed its Aug. 31 payroll, but after a valiant effort, it was processed in Philadelphia, just four days late, he says.
The next time disaster hits, there are not likely to be any missed payrolls, says Barron, who was appointed CIO in October 2005. Tulane now has a rapid-recovery contract with SunGard that guarantees capacity when needed in an emergency. The contract also provides for delivery of a mobile data center that could be used for local processing. And instead of moving backup tapes to a site in New Orleans once a week, tapes now go to Baton Rouge three times a week.
Leo Tran, Tulane's information security officer and the chief architect of the new recovery plan, says IT now has a detailed "directory of critical resources" -- hardware, software and people. The inventory can be used as a checklist before and after a storm to ensure that nothing is forgotten, he says. In addition to the more frequent tape backups, he says, every person in IT now has a USB key to which they can back up their own critical information so they can take it with them when a disaster threatens.
Coming up with better disaster preparedness and recovery plans was mostly a matter of hard work and attention to detail, Tran says.
Barron says no matter how well thought out a plan is, it should never be considered the final word.
"Every year now, in July, at the start of hurricane season, we sit down with the plan and see what we need to change," Barron says. "We go through all the scenarios again."
NEXT: F. A. Richard & Associates Inc.'s e-mail system zapped by hurricane