In February 2001, Gartner Inc. published a white paper titled "How Will You Get Your Data Back After the [insert catastrophe here]?" Seven months later, the Sept. 11 terrorist attacks filled in that blank, stunning the U.S. and forcing businesses to consider disaster recovery on a previously unimagined scale.
By 9 a.m. that day, as hundreds of thousands of workers passed through lower Manhattan on their way to their jobs, smoke was billowing over the city from the first hijacked plane's crash into the World Trade Center's north tower (see story). Within hours, the entire complex was in ruins, throwing Manhattan's transportation and communications systems into turmoil and the nation into shock over the vicious attacks on New York and Washington.
"There was a sudden awareness: We could have not just a building, but a regional catastrophe," said Dianne McAdam, an analyst at Illuminata Inc. in Nashua, N.H. "What happened with Sept. 11 in Manhattan, it was a loss of the phone lines, the data lines, the transportation, of parts of the entire communications infrastructure."
 |
 |
|
|
|
Even a buildingwide catastrophe was more than CareerEngine Network Inc. was prepared to handle.
The small, 3-year-old Web developer was located on the 21st floor of the World Trade Center's south tower, which was hit by a hijacked plane soon after the crash into the north tower. Several of the company's dozen employees would normally have been at work by the time the attacks began, but a series of minor emergencies and delays -- a sick child, a broken train, an apartment problem -- conspired to keep the staff out of the office and unscathed. CareerEngine's infrastructure, however, was demolished.
"We lost all our information about our customers. In some cases, we didn't even know who owed us money and who didn't. We had to contact all of our clients and have them work with us to re-create our records," said George Benoit, the company's chairman and CEO.
CareerEngine creates customized career information and job listings Web sites for Web publishers. Those Web sites, hosted remotely, remained live without interruption. But the company's internal data was wiped out.
"We had been making backups. And then we left them in the same place, in a different area," Benoit said. "We figured maybe there would be a fire, something like that we'd have to deal with. We never thought the building would fall down."
The dramatic events of last Sept. 11 galvanized some planners into action in a way that natural, more routine calamities like earthquakes and storms haven't. U.S. companies are starting to take disaster recovery as seriously as their European counterparts, said Ken Boyd, director of advanced disaster recovery solutions at IBM.
The 1993 Trade Center bombing, coupled with last year's attacks, created an environment here that resembles what European companies have faced for years from terrorist threats, he said.
"Before Sept. 11, we had a few [disaster recovery planning] calls a week," he said. "Now we are getting a few calls a day."
That has also been the case at data replication vendor NSI Software Inc. (NSI) in Hoboken, N.J. "The visibility of data protection and availability has been raised to an all-time high," said Bob Guilbert, vice president of marketing. NSI has been expanding the staff of a services unit it launched in October, an offering that quickly attracted several dozen customers seeking help constructing a business continuity plan.
Businesses are increasingly considering the possibility of a disaster that paralyzes a large geographic area, according to analysts and vendors.
"One of the lessons that customers learned from Sept. 11 in one sense is that it was a regional disaster -- not just a floor or a building, but all of the infrastructure around the World Trade Center," Boyd said.
The scope of the disaster undermined some companies' disaster recovery plans, as workers couldn't reach buildings to handle manual tasks and organizing phone calls couldn't be made. Consequently, IBM now stresses to customers the importance of automating data backup, Boyd said.
"You have to ask, How automatic is your disaster recovery?" he said. "How much are you depending on people to make decisions? Because you may not have a scenario that lets that happen."
Since the attacks, IBM has set up a technical design council that examines the disaster recovery proposals the company makes to its clients, checking that they will work in real-world conditions. IBM engineers examine the bandwidth connecting primary and recovery sites, for example, and evaluate hardware products to ensure those will work together as well in practice as they do in theory.
Customers are also extending the distances between data centers, realizing that a backup in New Jersey for a network in Manhattan could be vulnerable, Illuminata's McAdam said.
In the year since the attacks, customers in the New York metropolitan area have been adding "heavy lifting" data centers outside the New York, New Jersey and Connecticut area as an extra safeguard, said David Palermo, vice president of marketing for Wayne, Pa.-based SunGard Data Systems Inc., one of the industry's major providers of disaster recovery and business continuity protection services.
By the end of the week of the attacks, which killed three SunGard employees who were visiting clients at the World Trade Center, nearly 100 of SunGard's customers had placed the company on alert, and 30 had declared disasters (see story). The disaster declaration number was close to what the company fielded during the entire prior year. After SunGard bought the disaster recovery business of Comdisco Inc. last November and acquired its customers, those figures rose to 226 alerts, with 77 declared disasters (see story).
The severity of the Sept. 11 attacks prompted some businesses to escalate plans to implement or improve their protection plans, but, as is always the case, action lagged behind talk, Palermo said.
"It's the same thing as with Hurricane Floyd, with earthquakes, with any other disaster. It has the ability to make people say, 'Wow, we've been thinking about this forever.' But after any of those events, many more businesses think about it than actually end up doing something about it," he said.
For large enterprises, particularly those in heavily regulated industries, such as the financial services firms that fill lower Manhattan, fairly extensive emergency planning is a routine part and cost of doing business. Small and medium-size businesses are less likely to have plans in place, but even those that do face daunting obstacles in recovering from a catastrophe. Gartner estimates that two out of five companies that experience a disaster go out of business within five years.
Having a complete plan for both data backup and an off-site emergency work location for its employees allowed the New York Shipping Association Inc. (NYSA) to be fully operational two days after the Sept. 11 attacks destroyed its World Trade Center office. The 70-year-old organization, which coordinates cargo operations in the New York and New Jersey ports, drafted detailed plans after scrambling to recover from the 1993 World Trade Center bombing, said NYSA President Frank McDonough.
A SunGard customer, NYSA evacuated all of its employees safely on Sept. 11 and relocated approximately 100 of them to several temporary SunGard facilities in the region. Within the day, its staff had fully wired temporary workspaces, and data retrieval efforts were under way.
"I don't think there's a darn thing that we would have done differently. I've talked to the tech people a few times about this, and there's no second-guessing," McDonough said. After several temporary relocations, the company found a permanent new headquarters in March in Iselin, N.J., and considers itself fully recovered from the attacks.
But plans as comprehensive as NYSA's are expensive. SunGard has 15,000 recovery workspace positions throughout North America and Europe, but keeping such a physical office backup on tap carries a price tag that starts at several hundred dollars per month, per seat. Data backup can be less expensive -- low-end prices for a single server are a "couple hundred per month," estimates SunGard's Palermo -- but even those costs can be hard to fit into the budget for a small business.
"There was a lot of interest [after Sept. 11] and a lot of projects in planning phases, but many businesses have no appreciation of how difficult it's going to be to do," McAdam said. "This is not just throwing another backup at a remote location. The [small and medium businesses], some of them had a bit of sticker shock."
Sticker shock has so far kept CareerEngine Network from creating more extensive emergency plans, even though Benoit says he regrets not having a strategy in place before Sept. 11. While insurance and a $117,000 relief grant covered some of the company's losses, he estimates that his business lost $1.5 million because of the attacks. That financial hit, along with the depressed business climate of the past year, has forced CareerEngine Network to trim its staff from 12 to seven, and left it even less able to set aside money for disaster planning.
"One thing we emphasize to people is that disaster recovery is about recovering from an unplanned outage, while business continuity is having an infrastructure in place that supports running 24/7," McAdam said. "That's a very expensive infrastructure to build, but once you get it built, you really get disaster recovery for free. What we have to do is work with customers and get them to elevate their thinking, to think less about disaster recovery and more about keeping their business up and running 24/7."
Demand for high-availability products is soaring as companies realize their IT dependency, SunGard's Palermo said.
Ten years ago, disaster recovery was about retrieving mainframe tapes and rebuilding data centers. A 48-hour window emerged as the time a typical enterprise could expect to spend regrouping. But now, many businesses "can't go more than several minutes" without sustaining major damage, he said, and online storage systems and other new technologies are aiding them in creating failover systems and other backups to minimize downtown.
Tape will continue to serve long-term archival needs, but ever-dropping hardware prices allow customers to consider electronic storage strategies that would previously have been prohibitively costly, Guilbert said. He sees that trend, coupled with the increased awareness, as among the biggest post-Sept. 11 changes for the market.
"A year and a half ago, NSI would spend a lot of time [at shows] educating prospects. They'd say, 'What is replication? What does it mean?' That has clearly changed," he said. "Now we're definitely seeing a more educated CIO. I ask what sort of replication they're using now, and the answer is, 'We're not.' But they're trying to get educated on all the technology and solutions available because now they know they should be using something."
Ashlee Vance in San Francisco contributed to this report.
A year after the Sept. 11 terrorist attacks, has anything changed? What has your company done to better protect its assets? Should the government get more involved in preventing a cyber 9/11 in the future? Have your say in our Computerworld forum, 9/11 One year later.