Pilot fish gets a message from a user: There's an automated email she's supposed to get every day, and for the past couple days it hasn't shown up.
"I looked into it," says fish. "I soon realized nobody had been getting certain emails generated by the data warehouse system for the past three days."
He tests the email procedure from his PC, and discovers it only seems to work under certain combinations of From: and To:. It's definitely not something that's broken on fish's end, so he calls the help desk to open a trouble ticket.
"They told me to expect calls about this," the help desk tech tells fish. It seems the Exchange team has decommissioned the Lotus Notes server and switched all email to Exchange -- which has some moderately stringent requirements, such as that the From: field must be an actual email address on the system.
But for years, programmers have been putting whatever they liked in the From: field for the data warehouse to generate email reports. That worked fine with Notes. Now they're getting bounced.
"Even well-formed but no longer relevant email addresses were liable to cause much unnecessary network traffic, as they too got bounced around from server to server looking for a recipient who no longer existed on the system," fish says.
"Thus began an intensive two-week hustle to track down and correct every malformed or non-existent email address across two large mission-critical applications."
About 700 scripts need correcting. So do thousands of email entries stored across multiple tables in the two databases.
The Customer Orders Tracking team thinks it has corrected everything in its database, until fish writes a program to locate and alphabetize all email addresses -- and more that need fixing show up.
There are also dozens of email groups and aliases that have to be created or corrected. It turns out the Exchange team ported the email groups over from Notes -- in many cases incorrectly, mistyping the names. The scope of the fix just keeps growing.
Meanwhile, the Exchange team backs out its changes and revives the Notes server, so at least people can receive the automated emails again.
"Finally, the last patch went into production, and the email server cut-over took place again, this time with no issues," says fish.
"Six months earlier, when the Exchange team apprised the technical architects' team of the planned cut-over, none of the architects thought that it would impact their applications -- and they never said a word about it to anybody."
Say the word to Sharky. Send me your true tale of IT life at email@example.com. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.