Flashback about a decade, to when this retail chain's data center needs a sprinkler head changed for its fire-suppression system -- which turns out to be a little more complicated than expected, according to a pilot fish there.
"Back in the 1990s, we moved into a new headquarters with a state-of-the-art computer room," says fish. "Halon was on the way out, so they installed a 'dry' sprinkler system -- the pipes were pressurized with air, and would only fill with water when an actual fire occurred. This would prevent damaging nuisance leaks.
"And when there was leaking air and loss of air pressure in the system, we knew we needed a new sprinkler head."
The maintenance guy has done this several time before: First he closes the water valve to the system. Then he turns off the little air compressor that keeps the system dry by going to the clearly marked five-amp circuit breaker for the air compressor and flipping it to off.
And the entire computer room goes dark.
Not just the rows of minicomputers, storage arrays and racks full of servers, but also the three big air conditioning units that are now wheezing to a halt. It's 8,000 square feet of deafening silence.
"There were lots of open mouths, wide eyes and frantic running around," fish says. "Flipping the breaker back to on had no immediate effect. Luckily, it only took a few hours to get the room and systems back up and running again.
"It took a lot longer to answer the question, 'What happened?'"
And that, fish and his cohorts discover, goes all the way back to when the computer room was built. The plans called for the air compressor to be on its own circuit. But the electrician wired the fire-alarm panel to the same circuit -- so turning off the compressor's breaker also cut power to the room's fire-alarm panel.
But why wasn't it a problem until now? It turns out the fire alarm has a backup battery. Even if power is cut, the battery keeps the alarm panel on.
And what was different this time? It seems the maintenance contractor for the alarm panel wasn't changing the battery every year as he was supposed to. The battery was dead, and the alarm panel shut down.
And then? "When the alarm panel lost power, it dropped the 'keep-alive' signal to the Emergency Power Off circuit -- the Big Red Button," says fish. "And the EPO did what it was supposed to do: kill all power to the computer room.
"An electrician was hired to run a new dedicated circuit for the air compressor. And a new maintenance contractor was found for the alarm-panel service."
Sharky needs to find a new true tale of IT life every day. So send my your story at firstname.lastname@example.org. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.