The local high school where this IT pilot fish works has a network that's more complex than some medium-size companies -- and it's a real problem when a popular virtual application server crashes.
"It was toward the end of the day when one of the virtual servers that host a proprietary database application went down and wouldn't come back up again," says fish.
"Luckily for us, the server vendor is a Microsoft Gold partner, and had one of its techs onsite with us already for a completely unrelated project involving setting up a new virtual host server in another building."
The downed server isn't mission critical -- just very heavily used -- and fish and his cohorts would rather not have to restore the whole multi-gigabyte database from backup. So the Microsoft people are called, in the hope they can offer a quicker solution.
That's about the point when fish leaves for the day.
Next morning, about an hour after fish arrives at work, he receives an email message from the network manager, who may or may not have actually managed to go home the previous night:
We have come to the realization that the better option is to ignore the Microsoft engineers and take a different approach to solving this problem, as they didn't seem to be getting anywhere.
Microsoft's insistence on repairing the previous server was taking far too long. So this morning we decided to restore the application server from a backup to a separate virtual host machine. Unfortunately, this is a slow process, as there is a huge amount of data to transfer.
Many apologies for the long delay. I will keep you posted on progress.
Some four hours later the database application is back up and running on a new virtual host server -- the one that was being set up the previous day.
Unfortunately, the new host isn't really powerful enough for the size of the database the application uses. As a result, it's painfully slow -- but at least it's working, so the school's staff is satisfied.
"Microsoft never got to the bottom of the problem," fish says. "But a few days later we had the server rebuilt and back on its original, much more powerful host. Everyone was happy, and the network manager even got to go home on time that day!
"The new server was, however, subverted from it's original purpose to remain as a backup host on the off chance something like this happens again."
If it happens to you, tell Sharky about it. Send me your true tale of IT life at firstname.lastname@example.org. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.