Welcome to Murphy's World!

The backup generator for this data center kicks in about 12 seconds after the power goes out, says an IT pilot fish who works there.

At least that's how it's supposed to work. "The system starts with a battery and then kicks in the gas-powered system," fish says. "This powers emergency lights, some wall sockets and, of course, the entire server room.

"The generator can supply several hours of power, plus more if you keep feeding it gas."

Along with the generator, each server also has a rack-mounted UPS. But since the generator kicks in within seconds, they've never had to run for very long.

So when the data center lights go out early one morning, fish isn't worried. He counts slowly to 12. But nothing happens.

He asks a tech standing nearby to count to 12. The tech does. What didn't just happen? fish asks. Pause. Tech: "The generator didn't come on."

"Our main server admin hadn't made it in yet, and our manager hadn't either," says fish. "A checkup call to him confirmed it was an area-wide problem -- all the traffic lights were out. We didn't know how long it would last, and the individual UPS boxes were slowly draining.

"After 30 minutes, we started the process of shutting down noncritical servers, in the hopes the critical ones could stay up."

While they're doing that under the glare of flashlights, a bigwig stops by to ask how things are going. Fish and the tech quickly explain what they're doing and the time-sensitive nature of the task -- they're trying to get the servers down before the switches and router lose UPS power.

Bigwig asks if they want him to bring in a box fan, since with the air conditioning off it might get hot in the server room. No thanks, fish says politely -- thinking, Where exactly would we plug it in?

"We found out later that the backup generator was tested the week before with no problems," fish reports. "But when it was needed, the battery failed and the generator was unable to get started automatically."

And why wasn't it cranked up manually? "Our building supervisor had one of the only keys to get into the generator room," sighs fish. "He was stuck in an elevator."

Sharky's Law: If anything can go wrong, it will make a good true tale of IT life. Send yours to me at sharky@computerworld.com, and I'll send you a stylish Shark shirt if I use it.

Have a similar experience? Then add your comments to the daily Shark Tank blog!

Join Shark Bait

If you love Shark Tank, you'll have fun with Shark Bait, a place where you can really tear apart IT. Submit your own tech-related "baits" for points, or rate other baits. Leave comments, build your profile, and try to become one of the "big fish." Visit the Shark Bait site to get started.

Copyright © 2007 IDG Communications, Inc.

Shop Tech Products at Amazon