Welcome to Murphy's World!

The backup generator for this data center kicks in about 12 seconds after the power goes out, says an IT pilot fish who works there.

At least that's how it's supposed to work. "The system starts with a battery and then kicks in the gas-powered system," fish says. "This powers emergency lights, some wall sockets and, of course, the entire server room.

"The generator can supply several hours of power, plus more if you keep feeding it gas."

Along with the generator, each server also has a rack-mounted UPS. But since the generator kicks in within seconds, they've never had to run for very long.

So when the data center lights go out early one morning, fish isn't worried. He counts slowly to 12. But nothing happens.

He asks a tech standing nearby to count to 12. The tech does. What didn't just happen? fish asks. Pause. Tech: "The generator didn't come on."

"Our main server admin hadn't made it in yet, and our manager hadn't either," says fish. "A checkup call to him confirmed it was an area-wide problem -- all the traffic lights were out. We didn't know how long it would last, and the individual UPS boxes were slowly draining.

"After 30 minutes, we started the process of shutting down noncritical servers, in the hopes the critical ones could stay up."

While they're doing that under the glare of flashlights, a bigwig stops by to ask how things are going. Fish and the tech quickly explain what they're doing and the time-sensitive nature of the task -- they're trying to get the servers down before the switches and router lose UPS power.

Bigwig asks if they want him to bring in a box fan, since with the air conditioning off it might get hot in the server room. No thanks, fish says politely -- thinking, Where exactly would we plug it in?

"We found out later that the backup generator was tested the week before with no problems," fish reports. "But when it was needed, the battery failed and the generator was unable to get started automatically."

And why wasn't it cranked up manually? "Our building supervisor had one of the only keys to get into the generator room," sighs fish. "He was stuck in an elevator."

Sharky's Law: If anything can go wrong, it will make a good true tale of IT life. Send yours to me at sharky@computerworld.com, and I'll send you a stylish Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Now you can post your own stories of IT ridiculousness at Shark Bait. Join today and vent your IT frustrations to people who've been there, done that.  

Copyright © 2007 IDG Communications, Inc.

Shop Tech Products at Amazon