It's the late 1990s, and this big company uses Lotus Notes for mail. But one employee has a personal AOL account, and when he goes on vacation he forwards his business mail there to make sure he doesn't miss anything important, reports a pilot fish on the scene.
"Problem was, his AOL -- Amateurs On Line? -- account had a 100MB mailbox limit," fish says.
"Just at the same time that his AOL inbox filled up, a correspondent sent this user a message with an attached video file that was just under the 10MB attachment limit."
But fish doesn't know that at the time. All he knows is that the server team is getting calls from a lot of users saying they can't get any email from server Notes4.
Fish and his team check. Indeed, the server is not responding. Fish knows that happens sometimes with Notes, so he applies the standard remedy: He stops and then restarts the Notes service on the server.
Then to make sure it's fine, he sends a test message -- which promptly bounces.
"What the heck was going on?" says fish. "I quickly looked at the directory where the Notes databases were held and sorted by size. Our infractor's email file was first on the list, tipping the scales at just over 18GB, an enormous size for that time."
Fish takes the user's database offline and copies it to his own desktop to clear up the space on the server. That solves the immediate problem for the other users.
Then fish looks inside the problem user's Notes mail database, where he can see the rule the user set up to forward all mail to his AOL account.
And he can also see the thousand or so bounce messages in his inbox. The first one has the subject line "NSFW."
The rest of the messages, each one bigger than the last, were from Notes, which was forwarding each message to the AOL mailbox -- followed by AOL's response with a copy of the bounced message. Each time AOL rejected the message because the mailbox was full, it sent back a copy of the rejected message, so each time the message went through the loop, it grew by just under 10MB as another copy of the rejected message was appended to the Bounce message.
"This is what so quickly brought the server to its knees," fish sighs. "I removed the forwarding rule and left a voice mail with the infractor letting him know in exquisite detail what he had done, with a few chuckles thrown in.
"Shortly after this, the CIO came down with a directive to disable mail forwarding from corporate accounts."
Keep your story safe by sending it to Sharky. I'll file off the identifying marks from your true tale of IT life when you send it to me at firstname.lastname@example.org, and you'll get a stylish Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.