This communications company uses cloud technology a lot for its services, says a pilot fish who's on an end-to-end testing team for the company.
"Word came down that we had to verify that all of the cloud-based assets were being populated to our cloud provider correctly," says fish. "So I started researching webhooks and server back doors.
"Then I was told in no uncertain terms that the only way we could do this was by querying the assets the way users would -- using HTTP requests."
Fish spends weeks arguing against this approach, mainly because the company has several petabytes of data, so testing it all that way will take forever. But management says to just do it -- no more arguing.
And fish does. He writes a Python script to hammer away at the cloud resources. He also checks with IT to see how many I/O threads the company's network can handle. IT says to keep it under 500, so to be safe fish keeps it at 50.
His little script works fine, but as predicted it's going to take forever. New word comes down from management: Make it faster. Fish's modified version doesn't bother to validate the data in the files it pulls down -- it just checks the HTTP response code for each file.
The bosses are happy. Fish still isn't so sure this is a good idea, but he starts running it at night, as ordered.
Within a week, the script has validated a significant chunk of the cloud assets and found gaps in the data. The bosses are thrilled -- until the cloud provider calls.
Their message: Please stop whatever you're doing, because it's breaking our servers.
"Turns out that my repeated hammering of the servers was causing so much server traffic that it was being interpreted as a denial-of-service attack," fish says. "They were going to have to block us for the sake of their servers. For a while you could even see a giant red spot on Chicago on the cloud provider's DoS attack map."
And the fix for the problem? Reports fish, "After a lot of angry emails, we went with a webhook -- like I wanted to in the first place."
Sharky has always wanted your true tale of IT life. Send it to me at firstname.lastname@example.org. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.