IT Execs Race Against Time Along Gulf Coast

Users scramble to shore up systems as the new hurricane season starts

From his office window at the University of New Orleans, Jim Burgard can see construction crews working feverishly to repair the London Avenue canal, which was breached during Hurricane Katrina last August -- causing flooding on campus and contributing heavily to the catastrophic inundation of most of the Crescent City.

Burgard, who is the assistant vice chancellor for university computing and communications, and his staff have also been toiling to shore up the school's IT systems for the possibility of another big storm. Like many other users working on similar initiatives in the section of the Gulf Coast ravaged by Katrina, he had hoped to meet a self-imposed deadline of last Thursday. That day marked the official start of this year's hurricane season.

But Burgard now doesn't expect to finish all the work needed to overhaul and upgrade his business continuity and disaster recovery capabilities until the end of August.

The University of New Orleans is still running 75% of its mission-critical applications on 25 servers housed at Louisiana State University's data center in Baton Rouge. The school had planned to shift those applications back to its own data center by mid-April, but Burgard said it ran into delays in upgrading its air conditioning and uninterruptible power supply systems and installing a new generator powered by natural gas. The culprit: the difficulty of snagging scarce -- and pricey -- contractors to do the work.

"It didn't make a lot of sense to switch our applications back to campus and have to deal with outages because of the work we're doing in the computer room," Burgard said. His new goal is to complete the data center upgrades by the end of this month.

The servers at LSU will remain there even after the work in New Orleans is done, turning the makeshift Baton Rouge data center into a hot-swappable disaster recovery site. Burgard plans to mirror and replicate the university's data to those servers so they can take over processing if the New Orleans campus has to be evacuated again. He said it likely will take until late August to finish setting up the replication process.

Other IT managers whose operations were in Katrina's path also are still working to put stronger defenses in place. For example, since Katrina rendered the Gulfport, Miss., headquarters of Hancock Bank uninhabitable, the bank has been running its IT operations from a building that previously housed its training staff. Now Hancock is building a hardened data center, also in Gulfport but 10 miles from the ocean and at a higher elevation than the former headquarters, said Norman McDonald, manager of information security and business continuity at the bank.

However, the new data center isn't scheduled to be completed until next spring. "We will be facing the upcoming hurricane season in a less than ideal situation buildingwise," he said.

In addition, the bank -- which ran its IT systems from disaster recovery hot sites in Chicago and Atlanta for two and a half months after Katrina -- began a project late last month to set up capabilities for replicating data to the Chicago facility. That work should be completed within 60 days, according to McDonald. The goal, he said, is to have the bank's customer-facing systems back up within four hours of being shut down in the event of a hurricane or other disaster.

One of the main challenges for the New Orleans-based Ochsner Clinic Foundation during the aftermath of Katrina was its inability to generate enough power at times to cool its primary data center. The health care provider, which operates a 600-bed hospital in the city and medical clinics throughout Louisiana, has since added a fourth diesel generator and acquired some portable spot chillers, said CIO Lynn Witherspoon.

Ochsner also has tried to improve the redundancy of its WAN by adding a third Internet circuit in Covington, La., and burying an existing one that previously was located on a telephone pole on the hospital grounds, he added.

But there's more to be done. Although Ochsner has a disaster recovery hot site in place for its mainframe, Witherspoon said the foundation needs a recovery plan for its Lawson ERP applications and its electronic medical records system. The IT staff has begun a project to replicate the medical records to a data center in Baton Rouge, but that isn't scheduled to be finished until the end of the summer.

"Katrina was the hundred-year storm," Witherspoon said. "We'd all feel much more comfortable if some of this was in place right now because of the fragility of the hurricane protection here in New Orleans. But we're crossing our fingers and hoping this isn't the second hundred years."

That isn't a sure bet, according to the National Oceanic and Atmospheric Administration. In a report released in late May, NOAA said there is an 80% chance that this year's hurricane season will be another above-normal one. The agency predicted that there will be between 13 and 16 named storms this season, with four to six of them developing into major hurricanes.

Keesler Federal Credit Union in Biloxi, Miss., will complete a project to install satellite communications technology at its 14 U.S. locations by the end of June, said Larry Mayo, Keesler's vice president of information technologies. The credit union also has deployed a converged data and voice network that will be able to work over the satellite links.

However, the new setup couldn't handle all of Keesler's voice traffic if local telecommunications service went down again, Mayo said. "We're making a trade-off until we get our redundant data center outside the Gulf Coast built," he said, adding that the processing of voice and data traffic will be collocated at the new facility.

The remote data center will be located either in Atlanta or Dallas, according to Mayo. Keesler also plans to replicate data to that facility and to two locations in the Gulf Coast region, he said. Work on the replication project began in May.

Officials at Intralox LLC, a maker of modular plastic conveyor belts in Harahan, La., thought it was well prepared for a disaster before Katrina struck. But they quickly "learned we had a lot of other things to do," said Stuart Smolkin, the company's marketing strategy manager.

Now Intralox has set up a satellite assembly facility in Dallas and identified workers who will go there if another major storm threatens the New Orleans area. The company also has located a customer service group in Dallas to take orders. In addition, it has installed redundant hardware and software there and set up a fail-over system for routing phone calls to Dallas. "If we had to leave here, in a matter of hours we'd have all our major systems running," Smolkin said.

The day before Katrina came ashore, the IT staff at Tidewater Inc. drove two SUVs packed with an IBM AS/400 and Compaq servers from New Orleans to Houston, where the company had an office with high-speed Internet access in place. Tidewater had already made plans to have its IP addresses automatically fail over, and the relocated servers were up within 54 hours.

But that wasn't good enough for John Chaffee, IT director at Tidewater, which provides supply vessels and marine support services to the offshore energy industry.

Tidewater has since set up a second data center in Dallas, spending about $500,000 for new hardware that was installed there and at its existing IT facility. Data will be synchronized between headquarters and the backup facility over a network. If another storm forces the company out of New Orleans, "we will hardly lose a beat," Chaffee said. "We should be up in Dallas within a couple of hours."

Chaffee's IT staff was testing the data replication capabilities late last month as part of its goal of finishing the work by June 1. But Chaffee said the real "window of worry" is from Aug. 1 to Oct. 1 -- the height of hurricane season. He isn't particularly concerned that a storm will put a bull's-eye on New Orleans again. Even so, he said he doesn't plan to take any time off during that period.

Copyright © 2006 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon