Along the storm-ravaged Gulf Coast, six months after Hurricane Katrina struck, there’s a big deadline ahead for IT managers: June 1, the start of the next hurricane season.
As Mardi Gras was celebrated in the still-recovering New Orleans this week, more than a dozen IT managers in Louisiana and Mississippi said they are rushing to build more resilient systems, improve their companies’ communications capabilities and apply the many lessons they learned from Katrina.
There’s a particularly strong push to increase data backup capabilities, especially via online replication to other sites, and to set up wireless and satellite communications systems, the IT managers said. And the work is being done with a sense of urgency because of worries about this year’s hurricane season, even while many employers struggle to find IT staffers to replace those who left the region and haven’t returned.
Many of the people leading the efforts to reinforce IT systems and networks have already been through a lot, both professionally and personally.
For instance, the home of David Scripter, regional IT manager of URS Corp.’s New Orleans operations, was located near a breached canal in the city. “I lost my house and pretty much everything I own,” Scripter said, adding that at least he, his wife and their three young children were all safe.
From a business standpoint, the engineering services firm’s offices were inaccessible for about a month after Katrina struck on Aug. 29 and caused devastating flooding the following day. Because systems were out of reach, URS had to buy new hardware, especially PCs for its end users, Scripter said. The company set up remote offices in Baton Rouge, La., as well as Houston and Austin, and it took URS two weeks after the storm to restore its IT operations.
Now Scripter is back in New Orleans and working to make certain that the company can recover from another catastrophic storm within five days. He said his staff has installed a storage-area network (SAN) in New Orleans and is investigating how to mirror the data on the SAN to an off-site location in another state. URS is also installing a system that will replicate separate Oracle databases used in 3-D engineering applications to remote locations. “We’re trying to get a lot of this stuff in place by June 1,” Scripter said.
Larry Mayo, vice president of information technologies at Keesler Federal Credit Union in Biloxi, Miss., is working under the same deadline.
Keesler’s headquarters building lost its roof and most of its windows to Katrina, but the credit union’s data center was spared because it was located between two cement floors in the middle of the building, Mayo said. A generator worked throughout the seven days that the company was without electricity, and Mayo took steps to activate Keesler’s disaster recovery hot site in Scottsdale, Ariz.
But the credit union decided not to go live with the hot site because of the widespread failure of voice and data networks in the Gulf Coast region. “Without the communications leg of it, no matter which data center we worked out of, we were going to be dead in the water,” he said.
As a result, Keesler is installing satellite communications links as network backups at its 12 locations in the U.S. and three offices in the U.K., he said. The installations have been completed at three facilities, and the remainder are due to be finished by June 1. By then, the credit union also expects to have deployed a converged voice and data network that will replace an aging Cisco data backbone and a private branch exchange phone system. The converged network, which was expedited after Katrina, will be able to work over the satellite links, Mayo said.
The day after Katrina struck, Neal Hennegan, director of technology at insurer Gilsbar Inc., had to drive 60 miles from his office in Covington, La., to find a working phone so he could declare a disaster and authorize hot-site operator SunGard Availability Services to switch his IT operations to its Chicago data center.
Like many IT managers who experienced the storm and its aftermath, Hennegan can quickly list the things he did right and the things he has corrected. For instance, he was especially pleased that Gilsbar had tested its SunGard backup capabilities before Katrina struck, because that helped avoid start-up delays at the hot site. But Hennegan said he plans to send IT employees to Chicago in advance of any future storms “and empower them to declare a disaster without talking to us.”
Even at a company like Blue Cross and Blue Shield of Louisiana, which had invested heavily in business continuity capabilities, Katrina managed to expose issues. For instance, the health insurer used diverse routing techniques on its networks but wasn’t aware that at one point they all went to a common physical site in New Orleans, Mehaffey said.
At the University of New Orleans, IT workers weren’t able to begin returning to the school’s campus until mid-November, said Jim Burgard, assistant vice chancellor for university computing and communications. And it wasn’t until the start of January that all of the employees who remained on the IT staff after the storm were back on campus.
Even now, the university is still running its mission-critical applications at Louisiana State University in Baton Rouge on 25 servers — some taken from its own data center, and some donated by LSU. But the New Orleans school is installing an uninterruptible power supply and a new generator powered by natural gas at its own facilities, Burgard said. When that work is completed in about six to eight weeks, the applications at LSU will be shifted back to the campus in New Orleans.
Even so, the university plans to leave the servers in place in Baton Rouge and mirror data to them so that the LSU location can function as a hot site if another disaster occurs. That strategy is also geared toward eliminating some of the problems the university had retrieving its backup tapes after Katrina, he said. Although the school didn’t lose any data, it had to wait three weeks to access the tapes, which were stored on the sixth floor of a building in downtown New Orleans
Meanwhile, Burgard is still trying to replace eight IT workers — nearly one-fifth of his 45-person staff. A recent advertisement for a senior networking position that before the storm would likely have garnered about 20 r¿sum¿s netted just four, he said. In addition, he received only two r¿sum¿s for a senior database administrator slot.
Also working to fill IT positions is Ochsner Clinic Foundation, which operates a 600-bed hospital on its main campus in New Orleans and 35 clinics throughout Louisiana. Ochsner lost 23 workers from its 150-member IT staff after the storm, said Bill Saussaye, director of information services operations. The health care provider has managed to fill some application development jobs with local workers but is using contractors to temporarily fill positions in networking and communications, Saussaye said.
Ochsner is also aiming to complete several disaster recovery projects before June. One involves the installation of redundant Freon and chilled-water cooling systems in the company’s primary data center — a plan prompted by its inability at times to generate enough power to chill the water in the existing system after Katrina hit. Saussaye said he also is looking to set up a remote data center at least 100 miles away from New Orleans and hot sites for Ochsner’s distributed systems
Jan Rideout, CIO at Northrop Grumman Corp.’s Ship Systems unit, lost her Mississippi home to Katrina’s fury. She said that if there’s a bright side to the storm for her, it’s that the disaster prompted the Ship Systems operation — which has facilities in New Orleans — to accelerate a plan to replace networking cables with wireless technology.
Disaster recovery plans have to be flexible and must take an organization’s ability to respond into account, Rideout said. For instance, many plans assume that employees will be available after a disaster occurs, she said. But when the air conditioners in Northrop Grumman’s data center stopped working after Katrina, there weren’t any employees on hand who knew how to gracefully shut down the servers, she said.
That points to the challenges that Rideout and other Gulf Coast IT managers are facing as they scramble to shore up their systems, networks and data-protection mechanisms so their companies can withstand another major storm.
“There is not ever going to be the perfect disaster-recovery plan,” Rideout said. “You can’t write the perfect script, and the next time will be different. And I don’t think we can forecast and predict every combination.”