It's time for this company's customer-facing network to move from one colocation data center to another one across town, reports the pilot fish who's managing the transfer.
"With 100 servers, switches, load balancers, firewalls, etc., it was a complicated project," fish says. "And with over 400 companies being served by that network, we had to keep downtime to a minimum.
"We did the move over a weekend, shuttling the equipment by truck across town. Everything was reconnected in the new data center and tested, and appeared to be working by Sunday night."
But at about 11 a.m. on Monday, as customer traffic begins to peak, fish and his team start getting calls from customers about being disconnected.
They start troubleshooting, but can't isolate the problem. Things stabilize by that afternoon, but the next morning brings the same problems, and try as they might, fish's group can't pin down the problem.
One big reason: The diagnostics indicate that the problem is occurring in a different part of the network every time the diagnostics are run.
Meanwhile, there's increasing pressure to fix the problem. The president of the company is calling every half-hour to find out what's happening. He's no IT expert, but he assumes fish and his team don't know what they're doing, so he Googles the problem and comes back with a list of troubleshooting tips -- which, of course, fish and his people already did the day before.
Then he adds his own suggestion: "Move the servers closer together, because maybe the data is having trouble getting from one rack to another."
Grumbles fish, "At that point, it was all I could do to keep my network engineer from telling him where he could put his servers.
"The problem turned out to be two hardware failures that occurred at the same time, a faulty switch and a bad flash card in a load balancer, both of which were only being used when traffic rose to a certain level. That's why the problem appeared to keep moving around."
Sharky's always on the move, looking for true tales of IT life. Well, OK, actually I'm always at firstname.lastname@example.org. Send me your story and I'll send you a stylish Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.