Contractor pilot fish is working the swing shift at a software company, and part of the nightly routine is to perform a walk-through of the server lab.
"One night I found a UPS with a blinking amber LED," says fish. "General troubleshooting for this UPS model starts with a front panel self-test.
"When I pressed the self-test button, I noticed the area suddenly sounded unusually quiet, which prompted me to look up. That's when I noticed all the servers were powered off with no lit LEDs. Uh-oh.
"Then I noticed the server names included those three panic-inducing letters, SQL.
"I ran over to a console so I could remote into my workstation to confirm my fears. The build team was complaining about a major system outage.
"I ran back to the UPS to reseat all the power connections. That's when I noticed the rack also contained a fiber switch with connections going to the adjacent rack. Yikes! -- I took down one rack by loss of power and a second rack by loss of network connectivity.
"Once I reseated and powered up the UPS, I had power restored to the first rack. I then worked with the on-call database admin and we got the build system online.
"The next day I asked the lab manager about that UPS. His reply? 'Oh yeah, when that UPS was installed, it had its front panel cable pinched. When touched it will shut off.'
"I decided that day to avoid UPS diagnostics wherever possible!"
Reach out and touch the Shark. Send me your true tale of IT life at email@example.com. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.