It's the early 1980s, and this sysadmin pilot fish works for a steamship company where software development is more than a little decentralized.
Case in point: "Our traffic department developed a container/cargo receiving application on one of our midrange computers," says fish. "The traffic manager and programmer worked together on development, creation, testing and maintenance. The IT department was not in the loop.
"During rollout, an error occurred where two different records would end up with the same index key. Whenever you closed out the system, the last person out received an error message that duplicate keys existed and did you want to ignore or fix?
"The proper response was to ignore."
But one hectic night while a ship is loading, the wrong clerk logs out last and gets the message. Not knowing any better, he tries to be helpful and selects the "fix the problem" option.
That locks the database -- right in the middle of the ship loading.
Traffic manager remembers that the programmer said a system purge would resolve the problem. So, in the hopes of getting the system operational fast, he runs a purge.
What he doesn't know is that, before running the purge, the programmer must identify and mark the duplicate record for deletion.
"The system created a new database and copied all the old records except those marked as deleted," fish reports. "The last step in the process was to delete the old file and rename the new file name to the old file name.
"Unfortunately, when the copy process came to the duplicate record it asked if the copy should continue with the duplicate record. The manager said no, not knowing this would end the record copy."
The purge then goes to the last step and deletes the old file. Meanwhile, the new file has none of the records past the old duplicate record.
The good news: The application is no longer locked up. The bad news: None of the records for the ship that's loading are there.
That's when the traffic manager finally calls IT for help. Fish and his cohorts track down the programmer, explain the problem and determine that the only way to restore the data at this point is from the most recent backup.
Which the traffic manager hasn't done in several days.
Sighs fish, "The fix was to find all the original cargo receipt documents and re-enter four days' worth of data. This was done and the ship was eight hours late sailing, at multiple thousands of dollars per hour in late costs.
"When the dust cleared, the IT department was in charge of the application and backups were done nightly."
Sharky needs IT -- and your story. Send me your true tale of IT life at email@example.com. You'll get a stylish Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.