Sysadmin pilot fish is away from the main office when he gets an urgent call from a junior admin: A server has crashed hard, and fish needs to return and get it back up right away.
"After calming the junior admin down and talking with both him and a more senior PC tech, I was able to walk them through some diagnostics," says fish. "We determined that the problem was a bad fan in this small, high-density rack-mount server."
With some instruction from fish and a little help from the PC tech, the junior admin is able to pull the bad server out of the rack, take the cover off, replace the fan with one that's in stock, close it up and put it back into the rack.
Before he hangs up, fish makes sure all the bases are covered. Is everything back online? Yes. Are users logging into the application running on that server? Yes. Is there any need for fish to return to the main office? Nope.
And that's that -- until two days later, when fish is informed that the server's application has crashed again.
This time he's at his desk, and he remotes into the server to check for obvious and easy-to-fix issues.
Instead, he's stunned to see the server reporting multiple drive failures.
"On my way to the data center, I ran through all of the things that I knew of that might cause multiple hardware failures in a server like this," fish says. "Heat? Power? A virus?
"When I arrived at the data center, I heard the little server beeping insistently. I immediately recognized the problem, and shut the server down.
"I corrected the issue and restarted the server, restoring service to the users. Then I went back to the junior admin and PC tech and explained to them why it is bad to install a rack-mount server upside down."
Submit your own true tales of IT life to sharky@computerworld.com. If Sharky uses it, you'll snag a snazzy Shark Tank shirt!.
Have a similar experience? Then add your comments to the daily Shark Tank blog!
Join Shark Bait
If you love Shark Tank, you'll have fun with Shark Bait, a place where you can really tear apart IT. Submit your own tech-related "baits" for points, or rate other baits. Leave comments, build your profile, and try to become one of the "big fish." Visit the Shark Bait site to get started.