This health care organization's disaster recovery site suffers a catastrophic failure of some networked storage, reports a pilot fish in the loop.
"Initially the failure didn't affect anything, and we weren't concerned because nothing production-related was affected," fish says.
"As we began working with the vendor to replace various components proactively so the failures wouldn't spread, we accidentally took down numerous data stores that host some test SQL databases. This caused the VMs to crash.
"Within a few minutes of the storage being offline, we caught wind that our organization's website was not working," says fish. "We were blind to the outside world.
"Being that we are a health care organization and allow patients and clients to view their records, refill prescriptions and schedule appointments, mass panic ensued. We were certain that nothing at the DR site should be running anything production-related.
"So we started our analysis, and replied back that no production systems were affected. These are only test, development and non-production servers that would be affected.
"Then it came out that someone had published code from a test server to the website..."
Test your story on Sharky. Send me your true tale of IT life at firstname.lastname@example.org. You'll snag a snazzy Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.