The Storage Networking Industry Association recently hosted an industry summit, “The Future of Computing – The Convergence of Memory and Storage through Non-Volatile Memory (NVM).”
For a glimpse into the future, from some of the industry’s true thought leaders, check out the presentations [ZIP] and audio recordings [ZIP]. This view into the rapid developments in flash memory and other NVM technologies has given me a new-found appreciation for the concept of a software-defined data center (SDDC).
SDN and the implications for storage
Infrastructure managers who believe in the promise of software-defined data centers are beginning to see storage as the final piece of a puzzle that includes virtualization and SDN. However, this is only possible if the storage infrastructure itself can be separated, into:
- software that controls and manages data, and
- infrastructure that stores, copies and retrieves that data.
In short, storage needs to have its own control- and data planes, each working seamlessly as an extension of the storage infrastructure.
There are several reasons to separate the control plane and liberate the storage control software from the hardware. Here’s one: Software-defined storage allows offloading the computationally heavy aspects of storage-management-related functions—like RDMA protocol handling, advanced data lifecycle management, caching, and compression. The availability of large amounts of CPU power within private and public clouds opens all kinds of possibilities to both network and storage management. Those options were simply not feasible before.
With more intelligence built into a control plane, storage architects are now able to take full advantage of two major changes in the data plane.
1. Optimizing performance for non-volatile memory
The first change involves advancements in NVM technology—both the increasing affordability of solid state memory such as flash, and the new capabilities promised by next-gen storage technologies such as PCM and STT-RAM.
Phase Change Memory (PCM) and Spin-transfer torque random-access memory (STT-RAM) have the access speeds and byte-addressable characteristics of the DRAM used in today’s servers. But, like flash, it also has the transformational benefit of solid state persistence.
These prototype technologies are hugely more expensive than flash is today, but it is predicted that one of them will, , surpass even the cheapest forms of flash memory. But don’t ask me which horse to back!
Regardless of which technology wins, the trends are clear: Within a few years, the majority of a server’s storage performance requirements will be served from some form of solid-state cache storage within the server itself. When this is combined with new network technology and software that thrives in a distributed architecture, it has major implications for storage design and implementation.
Imagine how your infrastructure would change if every server had terabytes of super-fast solid-state memory connected together via ultra-low latency, intelligent networking. The fact is that many of the reasons we implement shared storage for mission critical applications today would simply disappear.
Apart from niche applications, this vision is still a long way off, but this is where our industry is heading.
2. Optimizing capacity for really large disk drives
The second major change is the demand to store and process massive amounts of data, which increases as we are able to extract more value from that data through Big Data analysis. This coincides with a dramatic reduction in the cost of storing that data.
Very high density SATA drives, with capacities in excess of 10 TB per disk, are coming. But in order to surpass some hard, quantum-physics level limitations, they will use new storage techniques—such as shingled writes—and will be built optimally to store, but never overwrite or erase data.
This means the storage characteristics at the data plane will be fundamentally different from those we are familiar with today. Furthermore, even with these improvements in the costs and density of magnetic disk, the economics of power consumption and data center real-estate mean that tape is becoming attractive again for long term archival storage.
Finally, think about the economies of scale that large cloud providers have and the availability of the massive computing power they are able to place in close proximity to that data. This means those cloud providers will have a compelling value proposition for storing a large proportion of an organization’s cold data.
Regardless of where and how this data is stored, the challenges of securing and finding that data, and managing the lifecycles of this massive amount of information means traditional methods of using files, folders and directories simply won’t be viable. New access and management techniques built on top of object-based access to data such as Amazon’s S3 and the open standards based CDMI interfaces will be the management and data-access protocols of choice.
In the end, the only way to effectively combine the speed and performance of solid-state storage with the scale and price advantages of capacity-optimized storage is by using a software-defined storage infrastructure.
It is the intelligence of a separate control plane powered by commodity CPU that will allow infrastructure managers and data center architects to take advantage of these two massive trends.