Shark Tank: It never rains, but it pours

This company creates a new division for a hot new technology, and to support it, it builds a brand-new building for its call center, reports an IT pilot fish working there.

"The 24/7/365 call center had strict uptime requirements, but the data center had some design flaws," fish says. "This was the first data center I ever witnessed to have an entire wall of floor-to-ceiling windows looking to the outside world.

"Strict uptimes are difficult when a storm breaks a window and it rains in your data center.

"But upper management wasn't too concerned with the odds that severe weather could cause water damage and days of downtime if a window got broken, so I waited quietly for the I-told-you-so day."

And, months later, it comes: Fish is awakened in the middle of the night by a phone call. The call center is down, he's told, and it's raining in the data center.

But driving in, fish suddenly notices something unexpected. It's a clear night, with no storms in sight. Where's the rain coming from?

"I made my way to the data center to see fans positioned in the doorways, blowing hot air out of the data center, and a janitor attempting to mop up the water that was pouring out of the ceiling tiles just feet away from the server racks," says fish.

"Postmortem showed that a lengthy power outage had caused the UPS and generator fail-overs to kick on. All systems functioned as designed, but a pump in the condenser drain for the main air-conditioning unit was not wired to the fail-over power.

"As the cooling system ran, the condenser pan overflowed, causing the 'rain.' The overflow eventually triggered a float that shut the air conditioning system down, causing the heat."

In the end, it turns out that many of the servers and telephony equipment didn't fail because of water damage; it was the heat that did them in.

But management is now convinced that the water hazard is real. Fish gets the job of having the pump rewired -- and then wrapping the entire server room in plastic, "just in case."

"I spent the next week rigging sheets of plastic and duct tape to protect server racks and the UPS while maintaining air flow," fish says.

"The week after that, I posted my resume on an online tech job site. Two weeks later, I submitted my resignation, with plastic still in place."

Submit your own true tales of IT life to If Sharky uses it, you'll snag a snazzy Shark Tank shirt! You can also add comments by using the form at the bottom of this page.

Copyright © 2006 IDG Communications, Inc.

Shop Tech Products at Amazon