Last November, a fire broke out in one of the buildings on ISTA Pharmaceuticals Inc.'s main campus, forcing about 50 employees to move to another location on the property. After the building's sprinklers kicked in, the entire network had to be shut down because the water threatened the equipment carrying the company's inbound data traffic.
Managers and employees at the Irvine, Calif.-based ophthalmic pharmaceutical company handled the situation with composure, says IT Director Keith Bereskin. The company's network and core applications were back online within two hours, and only 10 of the affected employees had to stay away from their offices for more than three hours, according to Bereskin.
Not bad, "considering it wasn't something we formally talked about," he says.
ISTA's "mini-disaster" happened to coincide with a disaster recovery gap analysis being conducted at the company. In that analysis, a consultant discovered that the IT department, which oversees disaster recovery coordination, and the business divisions needed to communicate more effectively, says Bereskin.
The fire and the subsequent analysis helped spur ongoing discussions between Bereskin and his peers in various business departments to determine what their expectations would be during a recovery. Among other things, they're working to identify the data they would need right away and the systems and processes that would have to be restarted immediately.
"People have said to me that we should have had these discussions previously," says Bereskin. Now they do; Bereskin says he coordinates disaster planning meetings with his business peers several times a year.
The situation at ISTA highlights the types of communication problems that often exist among disaster recovery managers, business executives and line workers, according to disaster recovery experts. "The people side of disaster recovery planning is often overlooked," says John Linse, director of business continuity services at EMC Corp. At many organizations, when it comes to communicating disaster recovery plans, "there's almost this 'shoot, ready, aim' kind of approach," says Linse.
For instance, one of EMC's Midwestern customers didn't have an effective disaster recovery plan in place when it suffered a power outage last June, so a $7-an-hour security guard ended up being the one who made the decision to send home the 1,300 affected employees. The outage lasted two days and cost the company $1.3 million in business, including estimated lost revenue for orders that couldn't be taken. Afterward, EMC helped the company craft a business continuity plan that included identifying key business processes that need to stay up during a disaster -- and which people are responsible for them.
Tool Time
At Austin Energy, CIO Andres Carvallo says the purchase of a disaster recovery planning tool was an essential element in bringing key decision-makers together to craft a recovery plan in late 2003. Using the Living Disaster Recovery Planning System (LDRPS) from Strohl Systems Group Inc., Carvallo and Austin Energy's disaster recovery manager worked with supervisor-level business process owners to identify which processes needed to be recovered and when.
"As you go through this business by business, you populate the software with business processes and the people who need to be involved in the decision-making," says Carvallo. "In our case, 1,600 people are impacted by the tool."
Although LDRPS is only one component of Carvallo's effort to communicate the disaster recovery plan to his fellow Austin Energy employees, he says it has played a big role in helping the utility map a strategy and get the message to resonate with its staff.
Since Austin Energy deals with power outages on a regular basis, disaster recovery is already embedded into its culture, but Carvallo says that prior to his arrival at the utility in early 2003, business continuity "really wasn't understood as a responsibility of every line of business. So we had to drive this companywide."
LDRPS has helped Carvallo achieve that goal because it can track the percentage of the disaster recovery process that each manager is responsible for. "It helps drive this whole notion of accountability," he says.