Shark Tank: Paging Dr. Murphy

This health care company's data center houses a mainframe, database servers and wide-area networking gear serving five hospitals, so it's got to keep going around the clock, says a network pilot fish who works there.

What if there's a power outage? "We have a large UPS system that keeps the entire building up and running long enough for a large diesel generator to start up," fish says. "That, in turn, will power the computer room floor until outside power is restored."

Enter a new operations manager, who decides to test that system during the day. He checks and rechecks everything he can think of to make sure it will all work to spec. He announces to the entire company when the test will happen, but assures everyone that it won't cause any problems for users.

"Finally, the day arrives for the big test," says fish. "Power is cut to the computer room, the UPSs do their job and the diesel generator roars to life. Everything works just as expected, and we all breathe a sigh of relief.

"Until the generator suddenly gasps and sputters to a halt.

"The entire data center goes dark. The mainframe loses power. Everyone runs outside to see what has happened."

Turns out the operations manager hadn't quite checked everything. "He never thought to check the fuel level on the generator," fish says. "It had enough fuel to start and run for about five minutes. Then it ran out of gas."

It takes the mainframe group 13 hours to recover from the power outage, which is officially attributed to "unforeseen circumstances surrounding a test of the backup power system."

And fish's own network servers? "Two months before, when this operations manager asked that we remove the rack UPSs from the servers in the computer room because the entire building was on generator backup, we refused, citing experience with Murphy's Law," says fish.

"Our servers never went down.

"I think the operations manager eventually got a promotion..."

Copyright © 2004 IDG Communications, Inc.

  
Shop Tech Products at Amazon