There's caution, and then there's caution

A very large thunderstorm is headed for this company's data center -- an unusual event that management isn't quite ready for, says a pilot fish working there.

"When several similar companies merged in the late '90s, the one with the newest data center was chosen to be the production data center," fish explains. "However, it was originally built for a much smaller company than the merged entity, so it had to be immediately expanded.

"In the intervening years, the quantity of servers and network and disk hardware in the data center increased rapidly, but the UPS and emergency generator didn't keep up with the growth."

So with a big storm on its way, one high-level manager overrides the advice of the technical support team and decides to switch the data center over to emergency power, rather than risk the possibility that the city power could be disrupted.

The emergency generator is started up and the switchover is made on a Friday evening without a hitch.

But it turns out the generator has never been tested since it was installed -- and the diesel fuel it runs on had been sitting in the tank for a long time, so sludge has accumulated in the bottom of the tank.

When the sludge reaches the fuel line, the engine sputters to a stop. Shortly after that, the underpowered UPS runs out of juice.

And the entire data center shuts down catastrophically.

"The important servers and network were back up and running within 24 hours," sighs fish. "But some of the large databases took days to recover due to data corruption.

"And by the way, the city power never had an interruption."

The Shark Tank runs on your true tales of IT life. Send me your story at sharky@computerworld.com. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Now you can post your own stories of IT ridiculousness at Shark Bait. Join today and vent your IT frustrations to people who've been there, done that.

Copyright © 2009 IDG Communications, Inc.

  
Shop Tech Products at Amazon