Backups Gone Badly

Computerworld -

Late one Friday night, while working with the database administrators to perform an upgrade and backup, a typical Colorado Springs storm blew in. The data center walls trembled, then came a huge bang, the lights flickered, and a message on the Univac read, "Site Terminated." The standby generator and storage batteries were housed in a small building near the main computer center, so we went to investigate. A lightning bolt had ripped a 2-inch hole through one of the storage batteries. Our backup policy never anticipated that!
-- B.J., San Jose

When the regular IT technician was out sick, his supervisor decided to "clean up" some drives -- deleting our database of reports. It took our regular technician three hours to restore them from the backup. We had a NAT [network address translation] fail, but the backup NAT was OK, so the technician changed the name of the backup to be the same as the main NAT. That worked fine until the primary NAT was back online. During the nightly backup, the primary NAT, which was unused for 48 hours, backed up to the backup NAT -- erasing all files for the two days it was down. That technician was fired.
-- T.G., Baltimore

While I was on vacation, a young IT staffer, new to the company, decided to run his own "updated" version of an Oracle backup -- wiping out the live database. I had to fly back from Scotland to rebuild the database and restore from backup. Thankfully, only a half-day's work had to be input -- by 30 people!
-- L.B., Coral Springs, Fla.

We lost two years' worth of data at one site because of a former technician's mistake. The technician set up home folders for users for backup, mapped them to a file server, then removed the file server from the domain. Problem was, he didn't tell anyone that he removed that file server, and the log-on script he created for the users hid the error.
-- C.H., Portola, Calif.

Here's some advice about Linux Red Hat and Windows 2003 Server. When installing mixed operating systems, buy the required backup software licensing before you start! After our Linux server installation, our CEO didn't want to spend money to upgrade the backup software. So I resorted to incrementally and fully backing up to my workstation via a batch file. I quickly ran out of space and removed the oldest backup files. Two days later, those "old" files are just what our engineering department needed. What a loss. That same day, accounting agreed to purchase the additional license.
-- C.L., North Tonawanda, N.Y.

See the main story, How to Back Up Your Data Center.

See the complete Ultimate Backup Guide special report.

Do you have a similar story to tell? Share it at the Sound Off Blog..