Pilot fish arrives at work one morning to find a severity-2 trouble ticket on his plate -- and it's a weird one.
"The ticket stated that a Linux box in the DMZ was having a performance issue," fish says. "The user couldn't log in, and it was a production server, according to the user, who is in director-level management.
"That's strange, I thought -- I don't recall that this box made it through to the go-live stage. The monitoring wasn't even activated yet, which is why we didn't get any alerts."
Fish's first step, as usual: Log into the server with the root login. His attempt fails.
He tries again. It still fails. Fortunately, the network uses Kickstart for installing Linux remotely, so fish is able to use the Kickstart server to get access to the locked-out machine indirectly.
Now he can see why the machine isn't functioning. Turns out the log is full of failed queries from the domain name server. Why is that even running? fish wonders. He checks the process list and discovers that the box is also set up as a file server and print server -- none of which is in the deployment template. And the time stamp on the password files has changed, too.
This all looks awfully suspicious, especially on a DMZ machine that was originally built as a test server. Then fish remembers that the user had originally requested full root access, saying he wanted to install software that only he would have access to.
Since this user is a director, fish is very careful in how he crafts his email responding to the original trouble ticket:
After initial investigation, we found some suspicious services running. Are these needed for your particular applications? Also, the root password was changed. These services are not usually needed for a server on DMZ and may cause security and performance issues.
Sighs fish, "The user responded that the domain name, file and print servers came as part of the web-hosting tool Virtualmin and are not necessary, and that the root password had to be changed to gain access to Virtualmin.
"After turning off all unnecessary services, everything worked normally. I left the root password alone and gave the whole story to my boss -- who just threw up his hands."
Throw Sharky your true tale of IT life. Send me your story at email@example.com. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.