This business-products distributor receives payments from customers through a bank's service -- which works almost perfectly, reports a pilot fish in the distributor's IT shop.
"The customer mails the payment with a list of invoices that they're paying to our bank," fish says. "Our bank opens the payments and keys the data -- check amount, invoice number, etc. -- then sends the data to us.
"We receive the data and use a batch process called 'autocash' to apply the payments to the customer accounts-receivable balances."
At least that's what the IT department believes is going on. What no one has told IT is that for one large customer who pays hundreds of invoices per week, the distributor's credit department is applying the payments manually.
Turns out this customer has a bug in its payment system that mangles invoice numbers. Since the invoice numbers aren't right, the autocash job is unable to post the payments to the individual invoices.
The work-around: All this customer's payments are labeled "unapplied," meaning the payments aren't applied to any particular invoice -- just the customer's overall balance.
And that works fine for 11 years, with IT none the wiser.
Then disaster strikes: The customer fixes the invoice-number-mangling bug. Customer notifies the credit department that the work-around is no longer needed, and autocash is finally allowed to do its job.
So where's the disaster? "The customer didn't actually fix the bug," groans fish. "What they did was keep the rightmost six digits un-mangled and then just put a zero in the leftmost position. The result was that some invoice numbers matched in A/R and autocash dutifully applied them.
"But our invoice numbers are seven digits long, and some of those invoice numbers didn't belong to the customer -- they belonged to other customers. The credit department had to find the incorrectly applied payments and reverse them. Fun.
"And that went on for three weeks before the credit department let us know there was a problem.
"The fix was only five lines of code, but it's amazing that this went on without incident for almost 11 years!"
What Sharky doesn't know can't make it into the Shark Tank. So send me your true tale of IT life at firstname.lastname@example.org. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
The Best of Shark Tank includes more than 70 tales of IT woe submitted by you, our readers, since 1999. Which all goes to prove, conclusively, that hapless users and idiotic bosses are indeed worldwide phenomena. Free registration is all that's needed to download The Best of Shark Tank (PDF).