The company this pilot fish works for is located in a big industrial park, and when it's time to add a redundant data center, management knows just where it should go.
"They put it next door to the primary data center," sighs fish. "No problem, we were assured, there's a big diesel generator for both data centers that will automatically kick in should power fail."
And of course it does fail when a summer thunderstorm knocks out power. Both data centers switch immediately to battery power until the big new generator takes over.
It never does. Operations staffers just manage to complete an orderly shutdown of the servers before the UPS batteries die.
What went wrong? Turns out that, except for initial tests when it was installed, the generator has never been tested to make sure it will actually come online in an actual emergency.
So a regular testing schedule is set up. The tests are performed on schedule. Everything works flawlessly.
And the next time the power fails, the generator starts up, right on cue.
It's smiles all around -- for a few minutes. Then the generator sputters to a halt.
It seems the scheduled tests didn't include making sure fuel levels were topped off at the end of the tests. A "check fuel level" item is added to the testing checklist.
And at the next power failure, the generator starts up with plenty of fuel -- and a fan belt breaks, causing the generator to overheat and forcing a shutdown.
"The fan belts were long past their replacement date due to all the usage during testing, plus time in service," fish groans.
"At that point, our maintenance staffers read the @#$%! manual and started to follow all the manufacturer's recommended maintenance procedures.
"Not surprisingly, the generator worked well after that."
Sharky recommends regularly sending your true tales of IT life to me at email@example.com. You'll get a stylish Shark shirt every time I use one. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.