Demystifying de-duplication

Computerworld - Data de-duplication technology has emerged as a key technology in the effort to reduce the amount of data backed up on a daily basis, which in many enterprises is growing at more than 100% every year.

For example, John Thomas, IT manager at Atlanta-based law firm Troutman Sanders LLP, was able to use data de-duplication technology to reduce the amount of data streamed from more than a dozen remote offices and thereby cut his backup window from 11 hours to 50 minutes. Thomas says his compression ratio for his backups run as high as 55:1.

Vendors have taken different approaches to the technology resulting in multiple distinct products that users should become familiar with in order to choose the flavor that best suits their environments.

Data de-duplication uses commonality factoring to reduce the amount of data either at the backup server or at the target storage device. As a result of enormous compression ratios achieved by data de-duplication technology, disk is becoming more attractive as a viable, online alternative to traditional tape-based backup. For example, people working at remote and branch offices need instant access to all the data and applications available at their company's headquarters. So IT shops typically set up remote mini data centers, with application servers, block and file data storage, backup tape and report printers, sacrificing administrative control. By utilizing data de-dupe technology, backups can be performed over the WAN using spare nighttime bandwidth, eliminating the need for tape at remote sites.

Greg Schulz, senior analyst at The StorageIO Group, says de-duplication technology mainly resides in the backup space, complementing traditional tape libraries with the purpose of lowering costs and reducing data.

The main benefit to de-dupe technology is that you're not seeing your virtual tape library fill up, and you're "not seeing your backup targets fill up as fast as it normally would," Schulz says.

De-dupe can be done at the target of a backup stream (the storage array or tape drive) or at the source of the data being backed up (the application server). Traditionally, de-dupe products had been used as a target for backup data, but Schulz says there is "a growing emphasis de-duping back on the server."

Target de-dupe products are generally used as part of a final repository for backup data. Most backup software today supports tape volumes, files or raw disk as targets. Target de-dupe products mimic a tape library  and support virtual tape libraries (VTL), or they can act as a network-attached storage file server supporting network file system (NFS) or Common Internet File System files. Target de-dupe technology can also work on raw disk supporting Internet SCSI or Fibre Channel logical unit numbers (LUN). Prominent target based de-dupe products are sold by Data Domain Inc., Diligent Technologies Corp., ExaGrid Systems Inc., FalconStor Software Inc., Quantum Corp., and Sepaton Inc. 

Today, de-duping data at the target is the leading method, but de-duping data at the source where the data is coming from is even more disruptive, and the benefits are far greater, Schulz says.

Source de-dupe products replace backup software used in a client/server configuration, where remote clients de-dupe data being backed up and only transmit unique data to the central server. This reduces bandwidth requirements considerably, according to Schulz. Some prominent source-based de-dupe products include Asigra Inc.'s Televaulting for Enterprises, EMC Corp.'s Avamar, Network Appliance Inc.'s SnapVault, and Symantic Corp.'s NetBackup Pure Disk.

In band or out of band

Another characteristic used to discriminate target de-dupe products is when data de-duplication processing occurs. Data de-duplication takes time to compute and find commonality in the data being backed up. To minimize the effect on backup performance, some vendors de-dupe data in the background. These de-dupe products buffer the backup stream to disk and then after the fact reduce its size via de-duplication. ExaGrid, FalconStor and Quantum provide target de-dupe products that do background data de-duplication.