IT Execs Fight Fatigue, Labor Shortages

Operations in New Orleans still far from normal, they say

NEW ORLEANS -- In the year since Hurricane Katrina caused unprecedented devastation, IT managers here have done much to shore up their systems and try to ensure that their organizations can continue to operate, no matter what roars out of the Gulf of Mexico.

Many have added redundant voice and data lines or satellite communications systems and replaced tape archiving with electronic data backup. Power-generation capabilities have been improved inside data centers, and some companies have dug wells to guarantee that they have reliable water supplies. New contracts have been signed with disaster recovery providers.

But dealing with Katrina's aftermath has exacted a financial and emotional price on many of the people who have reassembled their IT operations while trying to put their personal lives back in order.

Kevin Bassett, IT manager at Morris Kirschman & Co., indicating the height floodwaters reached outside his facility

Kevin Bassett, IT manager at Morris Kirschman & Co., indicating the height floodwaters reached outside his facility

Image Credit: Patrick Thibodeau"The hardest part of this is fatigue," said Kevin Bassett, who manages IT at Morris Kirschman & Co., a furniture retailer based in New Orleans. "We've been doing this every single day for a year."

For Bassett as well as many other IT managers interviewed by Computerworld in late August, it has taken an unrelenting effort to deal with the professional and personal challenges created by Katrina, which struck the Gulf Coast a year ago last Tuesday.

Bassett is a member of the National Guard who was called to duty just before Katrina struck and assigned to work in the Louisiana Superdome, where thousands of people took shelter and then waited for days to be evacuated. He runs the IT operations that support Kirschman's retail business from a 250,000-square-foot warehouse in New Orleans. After the levees along some of the city's canals failed, the warehouse became an island in a 6-foot-deep lake and was inaccessible, he said.

The inside of the warehouse, which is raised to accommodate delivery trucks, took on about six inches of salt water -- destroying all the wiring that connected the company's systems to the outside world. The copper wiring has since been replaced with fiber, and Bassett has hired redundant telecommunications and data services providers.

But while he works to help Kirschman move in new retail directions, Bassett continues to fight storm-related problems, in particular an unsteady power supply. "We have power outages all the time," he said. "That is probably our biggest hindrance right now."

What may be even worse, though, is what Bassett sees when he drives out of the company's gate. Kirschman's warehouse is in one of the few places in the devastated Ninth Ward neighborhood with any activity. Much of the rest is nothing but a stark landscape of destroyed houses.

Bassett also provided a tour of other neighborhoods where the devastation extends mile after mile. In St. Bernard Parish, just outside New Orleans, he pointed out his uncle's house, which was covered with water and now stands ruined in a deserted subdivision. Not too far away, where the waters didn't rise as high, his grandparents are rebuilding their house. "There is something that we can save in this, and it's going to be home again," he said.

"People are under a lot of personal stress," said Paul Barron, CIO at Tulane University in New Orleans. "There are people working much harder in their jobs here because there are just fewer people, even though we're advertising for [more workers]."

Tulane, which runs one of the larger data centers in the New Orleans area, has about 70 IT staffers but is 10 people short of what Barron said he needs. And things have changed drastically for many of the school's employees.

"A lot of people in the Tulane community and technology services lost everything," Barron said. Some suffered deaths in their families.

The situation in New Orleans remains far from normal, according to Barron. The city's population is only about half what it was before Katrina, and the available housing is expensive. That makes recruiting employees from outside the area difficult. "I desperately need some DBAs but can't find them," Barron said. He added that outsourcing could become a possibility for Tulane if the IT labor supply in New Orleans doesn't increase.

In addition to dealing with on-the-job recovery issues, many IT workers still have to contend with ongoing personal concerns, such as working with contractors to rebuild their homes and trying to resolve insurance claims. Some continue to live in temporary housing, while many others have simply left the region.

"There are more jobs than people," said David Erwin, CIO at Adams & Reese LLP, a New Orleans-based law firm that also has offices in Washington, Houston and six other cities. Erwin said companies in the New Orleans area are competing against one another for the available IT workers.

"We are in this situation where we're hiring each other's folks," he said. "It's a tough job market [for employers]."

Increasing Costs

Katrina also has made it more expensive to operate IT facilities in the region by compelling companies and other organizations to improve their disaster recovery capabilities.

For instance, Tulane has been spending about $12 million annually on IT, but addi¿tional disaster recovery costs will increase its budget by $500,000, Barron said. In part, the increase will pay for a new contract with SunGard Data Systems Inc.'s Availability Services unit, which is providing the school with a mirrored Web site, e-mail fail-over and data processing resumption capabilities.

Tulane could easily spend more on disaster recovery if it were to move from backing up data on tapes to doing online backups, Barron said. But, he added, the bandwidth that would be needed to support the electronic data transfers is too expensive for the school.

Another change for IT managers in New Orleans is that many supply chain partners from outside the Gulf Coast region are now asking them about their disaster recovery plans, making it more than just an internal issue.

Sam Canatella, operations manager at Louisiana Steam Equipment Inc., an 80-year-old maker of industrial equipment that's located a short walk from the Mississippi River in New Orleans, said the large energy firms that make up his company's customer base want more information from him about how to communicate with people in the event of another major storm.

Prior to Katrina, Canatella had installed a remote data backup system for Louisiana Steam's accounting applications at a facility in Houston that is connected to the company's main data center via a virtual private network. Over the next year, he plans to put in a new system that will enable data to be replicated among three sites: company headquarters, the Houston office and a facility in Mississippi.

Disaster recovery was once "a side item," Canatella said. But the damage and disruptions caused by Katrina made him realize that Louisiana Steam's data may be its most important asset. If the company loses goods stored in a warehouse, "I can call the factory and order more," he said. "But I can't replace the information in the computer."

When it's suggested to IT managers that New Orleans may have unique disaster recovery needs because of its location and low elevation, the response is typically a weary smile. They say Katrina's most important lesson is to have a contingency plan that takes into account the utter failure of everything.

Unforeseen Isolation

"What nobody anticipated was complete isolation for a week," said Don Chenoweth, CIO at East Jefferson General Hospital in Metairie, La., which borders New Orleans.

East Jefferson opened in 1971, six years after Hurricane Betsy tore up the region, and it was one of three area hospitals to continue operating during Katrina and its aftermath. It was built at sea level, which is relatively high ground, considering that some adjacent areas are 6 feet below sea level. The facility has 15 generators, redundant communications lines and its own water well.

Chenoweth pulled up photos on his computer showing the hospital completely surrounded by water after Katrina. The water literally came right up to some doors but didn't make it into the building. However, wind and water damage knocked out the electronics of the generator that was supplying backup power to East Jefferson's data center.

After an orderly shutdown on battery power, the data center was out of commission for four and a half days, Chenoweth said. External voice communications were also knocked out, although the hospital had Internet access for all but a 12-hour period.

One thing Chenoweth discovered -- and other IT managers said they found as well -- was that it was easier to successfully make calls on cell phones that didn't use the local 504 area code. Consequently, East Jefferson has bought a number of cell phones that use the area code for the state capital of Baton Rouge.

In addition, the hospital now has its communications lines connected to three separate BellSouth Corp. access points to provide triple redundancy. It has also installed a satellite connection, and Chenoweth said he is continuing to look at ways to strengthen communications, such as possibly using carriers that don't route lines through local connections.

"What we're trying to do is create a situation here where we have five or six ways we can communicate instead of just a couple," he said. "That was a huge lesson, I think, for just about anybody out here."

Children's Hospital in New Orleans also has installed satellite communications capabilities since Katrina, and it sank a well to boost its water supply, said Mike McSweeney, the hospital's IT manager.

In many cases, though, technology problems will be resolved long before the city itself is fully rebuilt. For the region as a whole, the IT managers interviewed here talked in terms of a five-year recovery time frame. But many said they're uncertain what New Orleans will look like in the years ahead.

McSweeney said that life is still difficult for area residents. "It's tense, and a lot of people are frustrated with insurance companies," he said.


Not too far away from Children's Hospital is Loyola University. Bret Jacobs, its CIO, regularly checks the progress of the neighborhoods that adjoin the university's campus.

In one neighborhood that Jacobs drove through with a visitor, there were contractors' signs at almost every house and workmen everywhere. But as he drove farther from the campus, keeping an eye out for the potholes that can appear overnight, the brown, rust-colored lines that show floodwater levels were still visible on many buildings, gradually rising higher.

In areas where the floodwaters were highest, fewer contractors were at work. But one house had a sign proclaiming, "We will rebuild." Pointing to it, Jacobs said, "There is a real sense of, 'We have to.' "

Copyright © 2006 IDG Communications, Inc.

Shop Tech Products at Amazon