HP BladeSystem Matrix

During the Big Dig, the city of Boston erected a sign saying, "Rome wasn't built in a day. If it was, we would have hired their contractor." That's a good way to describe the general state of affairs regarding the ideal of divorcing services from hardware and pushing server management away from the physical layer. HP's BladeSystem Matrix goes a long way toward realizing this ideal of an automated datacenter, providing a wide array of very useful tools and functions, but falling just shy of the lofty goal of truly hands-free datacenter service deployment. Of course, nobody else has reached that particular goal either.

Although Matrix is newly packaged, it's not accurate to portray it as a completely new product. It's built on the foundation of HP Systems Insight Manager, with a heaping helping of associated services such as rapid-deployment software (HP's RDP), Microsoft Active Directory, server virtualization (VMware, XenServer, or Microsoft Hyper-V), and hardware in the form of the HP BladeSystem c-Class blade chassis and HP StorageWorks EVA Fibre Channel storage framework. At the center of all these moving parts sits the new piece: HP Insight Orchestration.

[ Take InfoWorld's scrolling tour through service provisioning and management in HP BladeSystem Matrix. ]

It's probably best to think of Insight Orchestration as, well, an orchestra conductor, weaving a multitude of players into a coherent symphony. The sheet music for this particular piece is based on templates created via a drag-and-drop, Flash-based interface, and reference everything needed to build a single server or a group of physical or virtual servers, including all network and storage links. With the possible exception of Scalent's Virtual Operating Environment, nothing is as close to defining the automated or adaptive datacenter as HP's Insight Orchestration.

From the ground up

It all starts with the hardware. HP's Matrix product is built from existing HP hardware offerings, including the EVA4400 and BladeSystem c7000 blade chassis. In the mix are the usual Fibre Channel SAN fabric switches and Ethernet switches. However, the two network switches really don't play into the overall picture. This is possible due to the 10G Ethernet modules and the 8Gb Fibre Channel links present in the chassis. Essentially, each chassis has all the bandwidth it needs with these links, releasing administrators and the Insight Orchestration software from the onus of having to interact at the layer-2 level to provide VLAN assignments and such.

The hardware in my test lab consisted of two c-Class chassis with a total of five blades, two EVA 4400 SAN arrays, two 8Gb Fibre Channel switches, and an HP ProCurve 5406zl switch with four 10G links and a few gigabit Ethernet links. This was the core of the Matrix solution. On the side were a few ProLiant DL 360 G5s running Microsoft Active Directory, the HP ProLiant Essentials Rapid Deployment Pack (RDP) server, and the HP Insight suite, including the Insight Orchestration software. All this hardware was separated into two racks, each roughly half full.

The setup and initial configuration of the Matrix product is not for the faint of heart. You must know your way around all the products quite well and be able to provide an adequate framework for the Matrix layer to function. Fortunately, HP currently sells the Matrix fully assembled only, and when the racks arrive, an HP integration tech comes along to get the solution up and running, provide some training, and do basic integration with an existing infrastructure.

Test Center Scorecard

HP BladeSystem Matrix

Management (20%): 7

Performance (20%): 9

Reliability (20%): 9

Scalability (20%): 9

Interoperability (10%): 7

Value (10%): 8

Overall score: 8.3 (very good)

Insight Orchestration offers two main views of the servers within its purview: a logical view that displays physical and virtual servers together, and a physical view that displays virtual servers underneath the physical host header. Physical servers can be broken out into server pools, separated by any number of admin-defined criteria, such as Intel and AMD CPUs, blades that should be used for Windows servers, blades that should be used for VMware ESX hosts, and so forth. This allows admins to specify which hardware gets pulled into service for any given purpose. For instance, a BL460c server blade with dual quad-core CPUs and 32GB of RAM might be in the ESX pool, while a lesser configuration might wind up in the Windows pool.

The datacenter as art

Once the hardware is up and running, the interesting part begins. Using the Insight Orchestration Designer's drag-and-drop interface, templates defining individual servers and entire application suites can be constructed in just a few minutes, verified for compatibility with existing hardware and hypervisors, then published. Next, an admin with rights to that template can fire off a task to build it, and Insight Orchestration does the rest.

[ Datacenter automation and virtualization go hand in hand. See the Test Center reviews of VMware vSphere 4, VMware View, Citrix XenDesktop, and Microsoft's Hyper-V and SCOM Virtual Machine Manager. See Network World's review of VMware vSphere 4 and comparative hypervisor performance tests. ]

The nuts and bolts of how this works is important to understand. Using existing links to all the involved pieces (with some exceptions in the EVA storage management), Orchestration builds each server according to the template specifications. If one or more servers are run under VMware VI3, for instance, Orchestration will use calls to vCenter to define the server parameters, reference an existing VM template, and build the new virtual server automatically. If the server to be built is a physical server, links to the HP ILO management processor for that blade are used to boot the blade, and connections are made to the RDP server to signal the desired operating system and configuration requirements for that particular blade. The blade then PXE boots, hits the RDP server, and begins the build process.

All this happens in the background, with the admin basically pushing a Go button. As an example, I was able to build a new template for a Linux-based application comprising a Web server and database server, then have the servers configured and built about 20 minutes later. I could then repeat that process as many times as I liked, by simply creating a new service from that template.

A few manual steps are required for some builds. Because Insight Orchestration is still missing some links to the EVA storage management software, certain builds must be prepared -- with LUNs defined and activated by a storage administrator -- prior to building of a new server. You can speed up these deployments by predefining significant numbers of LUNs of the required sizes and flagging them with unique keywords to be used by the Orchestration Designer templates to allocate storage. Ideally, Orchestration would allocate the LUN on the fly when the template is used to build a new server, but for now the LUN must already exist -- and contain the appropriate keyword -- in order for the build to succeed. HP hopes to remove this manual step soon by providing better hooks into the EVA management.

Costing, leasing, and clients

I was quite happy to see that the concept of server and service costing has been integrated into the Insight Orchestration solution. When a template is designed, the server and its application can be assigned an overall cost, which is present to the users when they request a service build. Especially with virtual servers, VM proliferation is not a laughing matter, and assigning costs to those servers -- and obviously to physical servers -- can help reign in the number of VMs running on a particular infrastructure. When users see dollar signs, they tend to act more conservatively.

But there's also the option of creating a lease time on a template. In essence, this provides a fixed termination date for a service build. Thus, a working group could request a build of a particular template with a year lease. At the end of the lease duration, the servers will be automatically torn down and the hardware returned to the available pool. This isn't completely automated, requiring administrator approval before any server destruction actually occurs, but it does offer another method to help reduce the number of extraneous servers littered about the network.

As you might assume, there is a self-service portal available in Insight Orchestration that allows non-administrators to provision services from prebuilt templates. As with all other aspects of the Matrix solution, users can be assigned privilege levels allowing them to choose from specific templates or even design new templates without having full access to Insight Orchestration.

Assuming that the users are technical enough to grok the relatively simple interface, the self-service portal can remove IT admins from the server build process altogether, an option not to be taken lightly.

Push and pull

In addition to automating server build processes, Insight Orchestration hooks into HP Insight Manager's X2X migration tools, allowing admins to migrate servers from virtual to physical and back again -- even converting between hypervisor flavors with V2V migrations. I ran a basic test of this, migrating a VMware-based Windows Server 2003 virtual machine to a physical blade. The whole process took about 30 minutes, and aside from a few early licensing issues unrelated to the actual migration, it went off without a hitch.

[ Vizioncore vRanger Pro, DataCore SANmelody, Scalent V/OE, Stratus Avance, and Marathon's everRun VM marry high availability and disaster recovery to virtual server environments. See the Test Center's review. ]

There's also a tool called Capacity Advisor. This is roughly analogous to VMware's Capacity Planner and allows you to monitor power, CPU, RAM, network, and disk I/O on physical servers and create reports based on that data. And there's a function named Smart Solver, which lets you run what-if scenarios based on that data, to determine how best to collapse legacy workloads into a virtual environment or new physical model. This tool also provides forecasting, which attempts to calculate how certain workloads will increase over time, and other functions to generate warnings when limits are eclipsed.

Coloring outside the lines

One big question with any automated system is what happens when something goes wrong. In IT, you can always count on something going wrong. Someone will inadvertently create a VM outside of Insight Orchestration, or a hardware failure will hit in the middle of a build, or any number of other problems will rear their ugly heads. Thus, I set about to break a few builds and see how Insight Orchestration coped with the problems and what the recovery process entailed.

I threw a number of wrenches into the gears, even a few unintentional ones, and overall the system was manageable and recoverable in every case. In the screenshots accompanying this article, you'll note one or two status displays with a large number of Failed jobs -- most of those were failed on purpose.

In a few instances, things did get out of sync, such as when an automated VMware ESX build on a physical blade had an HBA hiccup during the final boot and began a reboot cycle because it couldn't access the disk. Per Insight Orchestration, status remained at 60 percent while this was happening, because the system hadn't completed its build. I canceled the job in Orchestration, then rebooted the blade. This time, the HBA attached to the LUN properly and the new ESX blade booted normally -- but Orchestration still thought the job had failed and did not mark the server as present and accounted for.

The fix for this took a few steps, including cleaning out the assigned LUN, reassigning the blade from the maintenance pool back into the ESX pool, deactivating and removing the logical server that was attached to that blade, and restarting the Orchestration job. The whole recovery process took several minutes and required visiting numerous parts of the management suite, but was not a terribly challenging task.

In another instance, I created a few VMs directly through vCenter rather than through Orchestration. The process to bring them back into the fold was trivial; all I had to do was run a discovery process from the Insight Orchestration console.

In most cases, it's hard to force Insight Orchestration to do something that can't be done. The templates and service creation jobs conduct extensive resource verification prior to starting anything, and Orchestration will issue warnings and fail the job if everything isn't where it should be. It's also very nice to see clear and concise error messages, logging, and auditing available right from the console.

[ Take InfoWorld's scrolling tour through service provisioning and management in HP BladeSystem Matrix. ]

What HP has done with the Matrix product is to build a wall with existing bricks, but the inner workings are not entirely hidden from view. This is both good and bad: good because it's necessary to be able to quickly and easily dig under the hood and fix problems, but bad in that there are still relatively mundane manual steps required to fully realize the potential of the solution. I hope that when all the hooks are in place, HP retains the relative openness of the solution to allow experienced admins to turn a few wrenches when necessary.