No good root cause analysis goes unpunished


This big manufacturer has operations across the globe, which means a software developer pilot fish who writes factory-floor applications has more than the usual challenges.

"Recently we began to have issues in one of our plants in another country," says fish. "The manager of the department sent out a long, disparaging email blaming all of the costly problems on my software because the enhancements he had asked for in the past were not working.

"And he included my manager and my manager's manager in the email chain."

Fish doesn't much like being blasted in front of his boss, but he responds by politely asking which systems at the plant are running his software.

Then he checks the logs that his software has generated on those machines -- logs that trace everything the application and the machine's operator are doing for months. Fish has learned from experience that the detailed logs can dramatically simplify troubleshooting problems with both the software and the humanware.

But once he has pulled down three months' worth of logs, it turns out neither human nor software operation is the issue.

"Armed with a good stiff coffee, I was able to show that the systems were running older versions of my software," fish says. "When a new log file is created my applications write their version number in the first line, and if the app is restarted I log the event and again write the version in the log as the app restarts."

So fish writes up his response, complete with screen shots of the log file.

And copies of old emails showing he sent the complaining department manager a newer version of the application that included his requested enhancements, with instructions on how to update.

And a copy of the old email response from the department manager, thanking fish for the update and promising to have the newer version installed that day.

"Was my boss pleased that I had shown our department was not at fault? No," grumbles fish. "I was sent to sensitivity training because the department manager who first blamed me for the issue felt my response -- showing that he had promised to have his technicians update the software -- was too harsh."

Sharky is a sweetness-and-light kind of guy to anybody who send him a true tale of IT life. So email your story to me at You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.

The march toward exascale computers
View Comments
Join the discussion
Be the first to comment on this article. Our Commenting Policies