At this big university hospital, an incompatible piece of switching gear has been intermittently stopping access to an important clinical system, says a sysadmin pilot fish trying to deal with the problem.
"It required emergency outages at least a couple times a month for several months," fish says. "Both I and the network tech knew that equipment incompatibility was the cause, and that the only permanent solution was to implement a network equipment change.
"Yet we could never get it done, because no one would authorize a planned outage. We both had a background in a hard-nosed, no-nonsense military organization, and used to shake our heads at how many times we had been called to repair this, but had been denied the opportunity to take a 15-minute to half-hour outage to permanently solve the problem."
After repeated unplanned outages, the unit puts in a call to another organization that provides support that's more oriented to PCs and terminals than big networks.
The outside tech starts moving things around in the wiring closet, then gets lost -- and finally calls for help from the network tech, who calls fish in to help in what is now a much more complicated repair job.
How long will it take to fix the mess this guy made? fish asks network tech. Tech's estimate: an hour or two.
As he speaks, a light goes on for both of them: The change they've been wanting to make for months can be done in 15 minutes, and will enable a bypass of the wiring closet mess.
"Now is the time we should do the upgrade, if we can get permission," says network tech.
"This is the best chance we may ever get to get permission," says fish.
They immediately call their respective bosses and explain that the only way to get the system back online in under an hour is to put in the upgrade.
And because none of the bureaucrats wants to be responsible for a longer outage, or to have to explain to their bosses how the system got messed up in the first place, both fish and the network tech get permission to go ahead.
"Fifteen minutes later the system was back up, with the new solution to the chronic problem in place," fish says. "That communication path never went out for us again.
"All it took to solve that recurring problem was for someone to come in and really mess things up so badly that there was no other way out but to do the right thing."
Do the right thing for Sharky: Send me your true tale of IT life at email@example.com. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.