This manufacturer gets the local telco to provide a Metro Ethernet network to connect seven local plants together, and it all seems to work fine, reports a pilot fish on the scene.
"We're consolidating storage resources at the main manufacturing plant, and backups for one of the smaller plants have been set up to run overnight across the MetroE on a new server at that plant," says fish. "Several projects are underway and the MetroE is supporting things nicely."
Then one day fish notices that the backups, which used to take about eight hours to complete across the MetroE, are now running multiple days.
Users start to complain about network slowness, and the local IT team scrambles to find answers and move resources and backups off the MetroE.
Soon the timeline for the problems is narrowed to a single day: the day the local telco doubled the bandwidth on the MetroE.
"Apparently, as a customer service, our telco rep discovered we could double our current bandwidth -- from 50 Mbps at main sites and 10 at smaller sites -- without additional cost," fish says. "This was done on the day that the troubles began."
But what to do? Fish's team discusses swapping out a router that's near the end of its life or doing some routing changes, but there's no evidence either of those would help.
Eventually fish's team tracks down one telco tech with experience in both provisioning and networking. Fish isn't completely clear on what the tech is proposing, but at least she has a plan and is willing to take action.
And once her changes are implemented on the telco's side, the backups are moved back to the MetroE and the first night they run well, with transfer speeds verified to be substantially better. Everyone is happy and fish's team is ready to celebrate.
That is, until they get the telco tech's email informing them that she lifted speeds on all plants for the MetroE to 100 Mbps, and the company will need to talk to a telco sales rep to bring billing in line with the change.
"Online inquiries into transfer speeds indicate that the current speeds are in line with 100 Mbps bandwidth, and they also reveal that our prior speeds were more in line with 50 Mbps at all plants than the 50 at some plants and 10 at others that we were actually paying for," says fish. "So by upgrading we actually suffered a 60 percent cut in bandwidth.
"Thanks, telco, for the complimentary 'upgrade.' Remind me next time to say no thank you."
Sharky wants to actually thank you for sending me your true tale of IT life at email@example.com, so I'll send you a stylish Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.