Opinion: Demystifying de-duplication
De-duplication can be applied anywhere there is a significant amount of data commonality
Computerworld - Of the assortment of technologies swarming around the storage and data-protection space these days, one that can be counted on to garner both lots of interest and lots of questions among users is de-duplication. The interest is understandable, since the potential value proposition, in terms of reduction of required storage capacity, is at least conceptually on a par with the ROI of server virtualization. The win-win proposition of providing better services (e.g., disk-based recovery) while reducing costs is undeniably attractive.
However, while the benefits are obvious, the road to get there isn't necessarily as clear. How does one make a decision to adopt a particular technology when that technology manifests itself in so many different forms? De-duplication, like compression before it, can be incorporated into a number of different products types. While by no means a complete list, the major options for our purposes include backup software, NAS storage devices, and virtual tape libraries.
Even within these few categories, there are dramatic differences in how de-duplication is implemented, with each offering having its own benefits. The scorecard of feature trade-offs includes the following:
- Source vs. target de-duplication
- Inline vs. postprocessing
- Global vs. local span
- Single- vs. multiple-head processing
- Indexing methodology
- Level of granularity
As with any set of products, these trade-offs reflect optimization for specific design or market targets: high performance, low cost, enterprise, SMB, etc. For more detail on the range of de-duplication options and their implications, you may want to check out my colleague Curtis Preston's Backup Central blog.
Until recently, one aspect of de-duplication that was generally unquestioned was its focus: secondary data, particularly backup. However, there are growing signs that this too is changing. In theory, de-duplication can be applied anywhere there is a significant amount of data commonality — which is why backup is such a good fit.
However, if we look around for more examples of high data commonality, one area that comes to mind is virtualized server environments. Consider the number of nearly identical virtual C: drives in a VMware server cluster, for example. Recently, NetApp has been leading the way among storage vendors in suggesting de-duplication for primary storage in these environments. In fact, it has been steadily expanding its support of de-duplication, initially offering the technology on its secondary NearStore platforms, then on its primary FAS line, and as of last week on its V-Series NAS gateways, where it can de-duplicate the likes of EMC, HDS, HP and other storage.
Of course, for many, this is uncharted territory and the performance and management impact needs to be better understood. But given the higher costs of primary storage vs. secondary, the potential to achieve a 20:1 savings in storage, even for just a portion of the environment, is quite tempting.
Read more about Networked Storage in Computerworld's Networked Storage Topic Center.
- Why Projects Fail CIOs are expected to deliver more projects that transform business, and do so on time, on budget and with limited resources.
- The New Business Case for Video Conferencing: 7 Real-World Benefits Beyond Cost-Savings This whitepaper provides insight into the value of video conferencing in today's business environment, and how organizations are using visual collaboration to find...
- Gartner Magic Quadrant for Client Management Tools The client management tool market is maturing and evolving to adapt to consumerization, desktop virtualization, and an ongoing need to improve efficiency.
- Audit Ready and Asset Optimized: The Solid Promise of an Intelligent Software Asset Management Solution In this paper Frost & Sullivan examines the benefits of enterprise-grade Software Asset Management solutions, and how these solutions serve as the convergence...
- LIVE EVENT: 5/7, The End of Data Protection As We Know It. Introducing a Next Generation Data Protection Architecture. Traditional backup is going away, but where does this leave end-users?
- On-demand webinar: "Mobility Mayhem: Balancing BYOD with Enterprise Security" Check out this on-demand webinar to hear Sophos senior security expert John Shier deep dive into how BYOD impacts your enterprise security strategy... All Networked Storage White Papers | Webcasts