This factory has been slowly upgrading its network from coaxial cable to Cat-5 and finally unplugs the coax from the final hub, reports a pilot fish on the scene.
Then it has to be plugged back in. "Due to our MRP system requiring coax cables, we were forced to install one hub to connect this one server to the rest of the network," fish says. "And of course, due to limitations of coax, we were forced to leave the entire loop of coax cable intact to ensure communication."
One day, a vendor's sales rep stops by the factory. His company has installed a PC on the network for ordering products -- nothing more than a connection to a Web page, and it's never given fish any trouble.
But the sales guy spots a note someone has left on the PC about needing to order certain parts. "He promptly does the transaction, and at the end he gets an error about not being able to update the database at that time and to try again later," says fish.
"So he assumes it's a connectivity issue, and he knows exactly which hub this PC is plugged into. He goes to this hub, which is mounted on a shelf 17 feet in the air. After finding a ladder, the sales rep climbs to the top and notices a dangling wire and can see clearly where it should be plugged in. He plugs it in and goes back to try his PC only to find it made things worse: Now he can't connect to his Web page at all."
Meanwhile, fish is suddenly flooded with calls from users who can't access the Internet, file server, e-mail or anything else on the network. The entire IT staff is trying to hunt down the cause: A virus? A server gone bad? Defective network hardware?
After 15 minutes of frantic searching, fish decides to power-cycle each hub to see if that helps. He turns the first 10 hubs off and then on, and there's no improvement. Then he gets to the one that's 17 feet in the air.
"I find a ladder, and as I'm climbing I can clearly see that the BNC connector for the coax is plugged in again," fish says. "I know that with the new line we ran, it's most likely creating a feedback loop and this is what's flooding the network.
"I unplug it, and I can immediately tell everything is OK.
"As I'm climbing down the ladder, this sales rep comes around and says, 'You need to make sure your connections are good and tight. That one up there was so loose it slipped out. I went ahead and plugged it back in for you, though. By the way, I'm having a database issue -- could you come take a look at it?' "