I'm always surprised that so many people still want to build their private and public cloud on logical units, served out by Fibre Channel SANs.
From my point of view, we should have ditched the LUN years ago: Storage management problems at scale created by the LUN presents some of the toughest IT challenges, and those problems are a direct result of the massive success of virtualization.
OK, so how did virtualization create so many problems? Unfortunately, the efficiencies brought by virtualization at the server layer weren’t matched by traditional storage approaches. To get the best out of your server virtualization infrastructure, you really need to use some kind of networked storage to hold the virtual machine images -- for most people, this means carving out chunks of space called LUNs from their Fibre Channel SAN and presenting those to the virtual servers.
Now SANs are great, but they were never designed for virtualization: They were designed to solve the problems of big-iron Unix servers as they took on the mainframe. As a result, they’re dedicated, expensive, often over-engineered, and rarely flexible.
The genius of server virtualization is that it takes all the complexity of a server and wraps this up into a few files, which you can run and manage in the same way you might run and manage a word processing document on a group share. If the physical machine you’re using breaks down, that’s OK: you just run that same virtual-machine image on a different machine.
That might sound obvious, but it’s a key concept -- virtual machines are stored as a bunch of files on a file system that’s shared by a number of servers.
This is where we start seeing problems with SANs.
The first problem: SAN LUNs weren’t really built to be accessed by more than one server, so virtualization vendors created special kinds of shared file systems on them. VMware calls this a VMFS Datastore, and Microsoft calls them Cluster Shared Volumes.
These are all great technologies, but regardless of whether you are running with Fibre Channel or iSCSI or FCoE they are all limited by the “queue depth” -- how many commands you can send to the LUN at any given time.
For a Unix system that has one application with lots of LUNs, that works out great. But for a virtualized environment with lots of applications on one LUN, you start running into problems. This is one of the main reasons people see performance problems in virtualized environments, and it’s also why most virtualization professionals generally recommend only keeping 20–30 files on one of these shared file systems. The bad news is that this is inefficient, and gets to be really hard to manage as your virtual machine count grows. (Also when you consider that each of these files is usually around 20–50 GB, this limits the size of the file system to less than a Terabyte.)
So what’s so bad about that ? Firstly, even with that rule of thumb, a single hungry application or server can chew up the queue. This can be managed, but this makes a storage problem turn into a virtual machine administrator problem.
The second problem: VM sprawl. It doesn’t take long to chew up a 1TB LUN these days.
Every time the server/VM team has to ask the storage team to carve out new storage there are going to be delays, which slows down the agility and efficiency of the VM team -- not to mention the expense and the frustration of going through an excruciating change-control process designed for big-iron Unix servers.
It’s exactly this kind of mismatch between traditional SAN arrays built for Unix and the needs of virtualization that stops the private cloud in its tracks.
For many of my largest clients, the solution has been to avoid using LUNs altogether, and instead use NFS to take direct advantage of the rich data management capabilities that an advanced file system provides.
In my next post, I take closer look at the benefits of non-LUN solutions when designing for the cloud.