August 1, 2005 (Computerworld) Moving magnetic tapes in and out of storage would seem to be the most mundane of IT functions. Indeed, companies have traditionally seen the transportation and storage of backup media as so routine that they have relegated it to non-IT personnel such as couriers or outsourced the job entirely. But that's changing now, following a rash of high-profile horror stories involving lost data that have been compounded by legislatures and courts that no longer buy the "the dog ate my tapes" excuses.
In February, Bank of America Corp. lost a tape with credit card information on 1.2 million customers. In April, Ameritrade Holding Corp. told 200,000 current and past customers that a tape containing confidential account information had been lost or destroyed in transit. Time Warner Inc. reported in May that 40 tapes containing personal data on 600,000 current and former employees had been lost en route to a storage facility. In June, Citigroup Inc. said that a box of tapes holding personal information on 3.9 million customers had disappeared on the way to a credit bureau.
And sometimes tapes go missing inside a company's four walls. In March, a Florida judge hearing a $2.7 billion lawsuit by financier Ronald Perelman against Morgan Stanley issued an "adverse inference order" against the company for "willful and gross abuse of its discovery obligations."
The judge cited Morgan Stanley for repeatedly finding misplaced tapes of e-mail messages long after the company had claimed that it had turned over all such tapes to the court.
In theory, there are straightforward ways to avoid these costly and embarrassing mishaps. But those measures, such as data encryption and backing up to remote sites via secure networks, have serious drawbacks, so it's likely that trucks full of tapes holding sensitive information will be roaming the roads for years to come.
 |  |
| Image Credit: Richard Downs | |
Driven in part by regulatory requirements, Xcel Energy Inc. in Minneapolis backs up data to tape "in terabytes per week," according to Mike Carlson, vice president of business transfer and customer value. The tapes are taken off-site and stored by Iron Mountain Inc., a Boston-based records management and storage company.
Asked if his company is taking any special steps as a result of the recent highly publicized tape mishaps -- Iron Mountain acknowledged that it lost the Time Warner tapes -- Carlson says, "We are actively working with them to ensure that it's not a systematic glitch that puts us at risk." Nevertheless, there will always be some risk of human error, he says.
Iron Mountain performs at a 99.999% level of reliability in its media transportation and storage operations, says Ken Rubin, executive vice president for marketing. "Over the past 50 years, we have honed a chain of custody and inventory control process," he says. "We have basically automated out of the process nearly all of the exposure to human error, but not 100% of it."
A tape goes through several distinct phases as it moves between Iron Mountain and a customer, and each step is recorded via bar-code scans, Rubin says. There are other protections as well, such as special security systems and alarms in the company's trucks. Iron Mountain recently completed an audit of all its facilities and processes and pulled from service a few trucks that failed inspection, Rubin says.
Iron Mountain offers service-level agreements, such as one that guarantees times for returning a tape requested by a customer. But the company follows the standard industry practice of limiting its liability to the value of the physical media in its possession, not the content of the media. "The fees that Iron Mountain and all the vendors charge -- basically pennies per tape per month -- are nowhere near what would be required to take on any more liability than just for the media," Rubin says. Customers could buy separate insurance for content, but few do, he adds.
Rubin says the "best and most practical" way to protect confidentiality is to encrypt sensitive data before it's written to tape. And, he advises, "make sure that your methodology for moving tapes off-site has the best chain-of-custody processes imaginable."
Carlson says he has looked into Iron Mountain's Electronic Vaulting service, by which backup data is automatically encrypted and sent over a network to Iron Mountain. But the service isn't cost-effective for the very large amounts of data Xcel Energy backs up, he says. Iron Mountain agrees that the service isn't practical for large backup needs.
Carlson says it's faster and cheaper to ship large amounts of data on tape via air or truck than it is to transmit it electronically. IBM runs a disaster recovery center on the East Coast for Xcel that would require eight hours to bring online. That's easily enough time to fly tapes there from Xcel's Colorado data center or from Iron Mountain, Carlson says.
Last year, nearly three quarters of 388 companies polled by Enterprise Strategy Group Inc. (ESG) in Milford, Mass., said they infrequently or never encrypt backup data written to tape.
In a report, ESG said it was surprised to learn that government agencies and big financial services companies are among the organizations least likely to employ backup encryption. "Bank of America did not encrypt its backup tapes and thus suffered an operations and public relations debacle, the costs of which may ultimately far exceed the cost and operational overhead of encrypting its backups," the research firm said.
According to ESG, companies spend far more on network perimeter security than on storage security. But the report said that "the onslaught of publicly reported security breaches and impending legislation will cause a profound change in security investment priorities."
According to Steve Kenniston, vice president for corporate strategy at Iron Mountain, encrypting backup data takes time, and with an explosion in data at most companies, the time windows for backups are already squeezed. Although encryption offers better data security, he says, it may adversely affect data protection -- that is, making sure backup data is available quickly and easily for recovery purposes.
Kenniston urges his customers to consider classifying data according to its function and sensitivity. For example, the most sensitive data, such as payroll records, might be encrypted and/or electronically vaulted, whereas other data might not justify the cost of those measures. But this kind of data discrimination isn't something IT shops have typically done as part of their backup processes, he says.
Rent-A-Center Inc., a Plano, Texas-based chain of 3,000 consumer-goods rental stores, produces 30 to 40 unencrypted backup tapes every day and turns them over to Iron Mountain. The company is now implementing a "stem-to-stern encryption process" based on 128-bit keys and hash signatures, which can be used to reveal whether the contents have been altered, says K.C. Condit, director of technical services.
"There is some overhead with encryption," which is why the company hasn't done it until now, says Condit. "There have been some technology concerns and some people concerns as well. But we are getting to the point that you really can't afford not to do it."
Meanwhile, Prince William County in Virginia is scrapping its tape backup system in favor of backing up data to disk over a secure network to a remote site owned by the county. CIO Masood Noorbakhsh says the goals are to decrease the time it takes to run backups and restores and to increase security. Because it's a private network, it won't be necessary to encrypt the data in transit, he says.
Church Mutual Insurance Co. in Merrill, Wis., produces about 10 backup tapes per day, and its employees move them to the basement of a bank two miles away. Using a company such as Iron Mountain would offer some advantages, says CIO Christopher Graham, but it would cost more.
Church Mutual typifies the many companies that have yet to join the embarrassed ranks of Bank of America, Time Warner, Ameritrade and Citigroup. "Management right now thinks that what we have in place is adequate," Graham says. "Nothing bad has happened yet, so why spend more money?"