And they do this every month?

It's time for the monthly servicing of this server room's air conditioning, so a sysadmin pilot fish meets the AC guys when they arrive.

"The vendor had sent a team comprised of people who had never visited our office before," says fish. "I accompanied them to the server room, then headed back to my seat in the adjoining room to answer a phone call."

But fish's heart stops when he hears the AC guys screaming for help.

He rushes back to the server room, and sees in an instant what's happened. They've opened a servicing outlet without making sure the valve was closed — and water from the high-pressure chilled water line is spraying all over the server room.

"Things got worse when we realized that they were unable to close the valve because of the water pressure," fish says. "With the help of a couple of co-workers, we started filling water into garbage bags."

But just as it looks like the situation is under control, one of the AC guys discovers that there's an electric outlet just below the line that's spraying water. And he discovers it the hard way: by getting himself electrocuted.

Power to the server room has to be cut, and all IT services go down — a nightmare in the middle of the day at a financial services company.

"After about 30 minutes of struggling, the building maintenance guys managed to close the water line," says fish. "It took nearly five hours to restore all IT services, and by then most of the employees had gone home. Fortunately, all side panels in the server racks were closed at that time, so none of the servers, routers or switches was damaged.

"Now we have installed a new AC unit outside of the server room — and no more chilled water lines run through the server room."

Chill out with Sharky. Send me your true tale of IT life at sharky@computerworld.com. I'll send you a cool Shark shirt if I use it.

Have a similar experience? Then add your comments to the daily Shark Tank blog!

Join Shark Bait

If you love Shark Tank, you'll have fun with Shark Bait, a place where you can really tear apart IT. Submit your own tech-related "baits" for points, or rate other baits. Leave comments, build your profile, and try to become one of the "big fish." Visit the Shark Bait site to get started.

6 tips for scaling up team collaboration tools
  
Shop Tech Products at Amazon