It's 9 a.m. when a pilot fish at this manufacturing company gets the bad news: The MPLS switching for the network has crashed hard.
"Connections between plants within the city were maintained thanks to our metro Ethernet connection," says fish. "But connection to the servers containing many mission-critical systems was gone.
"Phone calls were made and the cause was found: Apparently a water-main break occurred near a major fiber line and the line was marked incorrectly. One of the city backhoes snagged the fiber bundle when they were digging. Crews from MajorPhoneCarrier were called in and work ensued."
Meanwhile, fish and her team manage to get some connections working. A few of the plants have DSL connections, and VPN is used to connect a few people so they can get some critical work done.
But that all ends when the contractor for MajorPhoneCarrier cuts the fiber completely to begin resplicing it, killing the DSL and the metro Ethernet too -- which means all the plants are 100 percent down.
At 8 p.m. fish drives by the site of the break. It's still surrounded by trucks for MajorPhoneCarrier and its contractor, so fish knows the work isn't done. Fish and her team are still keeping in touch via mobile text messages and email, but there's clearly nothing more to do and by 10 p.m. she gives up and goes to bed.
Next morning, MajorPhoneCarrier has completed its repairs, but some intervention is required before all of the plants are properly connected again. As work proceeds, one of fish's co-workers sends a tongue-in-cheek text to the group: It appears that BigCellCarrier network is more stable than MajorPhoneCarrier. Texting is always up.
Reports fish, "I replied equally tongue-in-cheek with: Lol. Yes. Can we run our MPLS on a cell network???
"My cell phone buzzed a short time later and I assumed it was a coworker chiming in on the joke. Instead, it was a message from BigCellCarrier: Message not delivered.
"Yes, Big Brother, we know you're watching. Apparently someone needs to learn to take a joke, Mr. MajorPhoneCarrier."
Sharky wants a lot more than 160 characters, so send me your true tale of IT life by email at firstname.lastname@example.org. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.