Consultant pilot fish transfers a client to an offshore support team -- and shows the offshore guys how to bypass the security on a key server.
"This is useful if we need to hack the server to fix issues and keep it working," says fish. "However, I warned them that if they used this knowledge for anything apart from fixing the service, I would catch them and prove it was them."
Flash forward two years: The client supported by this new team suffers a major outage on a live server. The offshore team blames the application's developer and insists it's nothing they have done.
That looks likely at first glance. But fish is suspicious: He knows the developer being accused, and he's sure the developer wouldn't have made that mistake.
So when the client asks fish to investigate, he does. Nothing looks out of place, but there seem to be some commands missing in the logs, commands fish would expect to see for the work done by the offshore team.
"When teaching people ways around computer security, it's always best to keep a little knowledge back, just in case you have to investigate them," fish says. "The one thing I didn't teach them is that the bypass codes they used wrote a different set of logs, so I could still trace what they had done."
Fish eventually determines that someone on the offshore team did indeed use the bypass codes to edit the system logs. That's why the commands are missing from those main logs.
But the commands are right there in fish's backup logs. And it's clear not only what went wrong, but which individual tried to cover his tracks after the crash.
Fish reports his findings, and the culprit on the offshore team is removed from working on the client's systems.
"It was a shame, as he was a good worker and knew his stuff," says fish. "But he could not face confessing when he had made a mistake, and tried to blame somebody else."
Confess to Sharky. I'll file off the identifying marks when you send me your true tale of IT life at email@example.com, and you'll get a stylish Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.