Disaster recovery, redefined

This data center has had problems with its database servers for most of a decade, reports a pilot fish on the inside.

"A few years ago, one of the cluster servers failed completely, so the disaster recovery cluster became the production servers," fish says.

"Less than a year after that, those servers started having problems. One server started showing a corrupted hard disk. Database program files went missing, and replacing the disk didn't stop the disk problems.

"I thought anyone with any hardware experience would know what that means.

"Well, it finally happened one afternoon -- the server failed.

"Let that sink in: The disaster recovery server that we were using as production failed because the main production server failed a year before that.

"Luckily they found an old 'replacement part' to get the server back up the next day."

Sharky needs fresh true tales of IT life every day. Send me your story at sharky@computerworld.com. You'll score a sharp Shark shirt if I use it. Comment on today's tale at Sharky's Google+ community, and read thousands of great old tales in the Sharkives.

Get Sharky's outtakes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.

Related:
Download the 2018 Best Places to Work in IT special report
  
Shop Tech Products at Amazon