Given the number of blackouts, hurricanes and other disasters that have come our way over the past few years, many CIOs are wisely re-examining their disaster recovery strategies. Executive Council members share some of their tried-and-true methods.
1. Dedicate and empower staff
At the New York Mercantile Exchange, CIO Sam Gaer has dedicated a department within IT to manage business continuity planning and disaster recovery. Gaer ensures that the department's leader has access to upper-level management by running interference.
"You can't just set up a director and a department and let them run on their own," he says. "The CIO must pay constant attention to this department and set of resources."
2. Divide and conquer
In order to ensure business involvement in the development and maintenance of the business continuity plan, Martin Gomberg, chief technology officer at A&E Television Networks, has separated business continuity planning and disaster recovery into two initiatives, each with its own governance and goals.
For disaster recovery, the goal is technical recovery, and the plan is created and managed by developers and engineers. Business continuity's goal is business process stability, and that plan is developed -- in partnership with IT -- by business unit representatives.
3. Make sure the plan can stand alone
"When a disaster strikes, the staff who wrote the recovery plan may not be available to execute it," says Greg Smith, vice president and CIO of the World Wildlife Fund.
"You have to make sure your disaster recovery plan will work with or without the internal key people who developed it." If the director in charge of financial ERP applications wrote the plan, for example, ask the business intelligence manager to test the recovery.
4. Challenge the business
"If business unit managers tell me they need an application recovered quickly, but that application is not providing revenue generation or financial compliance, I will challenge those individuals to think hard about how long they can really go without that application," Smith says. The same goes for staffing an off-site facility during a disaster. Determining the right people to involve -- as well as the right services to recover -- is part of the negotiation process.
5. Align disaster recovery with application development
At A&E, the IT team incorporates disaster recovery into its application development processes. "We've developed an isolated test environment that enables full-time access and continuous testing of all systems and applications," Gomberg says. "Our business-continuity database includes a report on application-testing status, so we know when a system was last tested and whether it demands our attention to assure its performance in recovery."
6. Tabletop tests won't cut it
Regularly reviewing your plan on paper is important, Gaer says, but it is not enough. In addition to tabletop tests, Gaer semiannually springs mock disasters on his crisis management team, which is made up of staff and board members, who must set up a replica data center so that it is operational within a few hours.
"Given how busy our board members are, it is not easy to demand their participation in these tests," Gaer says. "But their participation is extremely important and is a testament to NYMEX's dedication to disaster recovery."
7. Try (and test) before you buy
When WWF's Smith was looking at a new technology for creating systems images (snapshots of the operating system disk and registry settings that allow for a relatively simple recovery process), not only did he employ a "try before you buy" approach, he actually used the product in a test at no charge.
"It was a real test: entirely off-site with a different network, firewall and environment," he says.
8. Hold postmortems and adjust
What you do with the results of the test is a critical part of disaster recovery planning.
"If you are recovering without third-party services, create an action-item checklist out of your review of what worked well and what didn't," Smith says. "If you are working with a vendor, document what went wrong and use that report to outline your expectations for the next test."
Editor's Note: The CIO Executive Council is a professional organization for CIOs. Its mission is to leverage the strengths of a large coalition of CIOs for the purpose of achieving change within our organizations and shaping the framework for the future of IT.
This story, "Eight best practices for disaster recovery" was originally published by CIO.