Enterprise cloud put to the test

The potential benefits of public clouds are obvious to most IT execs, but so are the pitfalls -- outages, security concerns, compliance issues, and questions about performance, management, service-level agreements and billing. At this point, it's fair to say that most IT execs are wary of entrusting sensitive data or important applications to the public cloud.

How we tested these cloud computing products

Archive of Network World tests

But a technology as hyped as cloud computing can't be ignored either. IT execs are exploring the public cloud in pilot programs, they're moving to deploy cloud principles in their own data centers, or they are eyeing an alternative that goes by a variety of names -- enterprise cloud, virtual private cloud or managed private cloud.

We're using the term enterprise cloud to mean an extension of data center resources into the cloud with the same security, audit, and management/administrative components that are best practices within the enterprise. Common use cases would be a company that wanted to add systems resources without a capital outlay during a busy time of the year or for a special, resource-intensive project or application.

In this first-of-its-kind test, we invited cloud vendors to provide us with 20 CPUs that would be used for five instances of Windows 2008 Server and five instances of Red Hat Enterprise Linux -- two CPUs per instance. We also asked for a 40GB internal or SAN/iSCSI disk connection, and 1Mbps of bandwidth from our test site to the cloud provider. And we required a secure VPN connection.

Rackspace, Terremark and BlueLock accepted our invitation. Amazon did, then did not and refused to communicate further. The services we tested were comparable in many respects. Rackspace Managed Private Cloud scored points for cost transparency, a solid administrative portal and good overall performance. Rackspace was the slowest in many portions of the tasks we needed them to complete, although, to be fair, we were making requests that were outside of their traditional sales channels. Terremark Enterprise Cloud delivered speed and the best administrative portal, and also offered the lowest cost. The BlueLock Virtual Cloud offered strong processes and good administrative support, but was the most expensive.

Over the course of conducting this test, we learned several things. First, a customer can expect to have an enterprise cloud deployed and up and running within a week after the selection process is complete. Second, all of the vendors delivered strong security and comparable performance, albeit with vastly contrasting management components.

And, we found that enterprise cloud services can be expensive. We also discovered that each vendor seemed "squishy" on overall pricing. Our recommendation is to not assume that the enterprise cloud route is automatically cheaper than buying and provisioning your own servers. Do a thorough cost analysis and make sure to pin down your vendor when it comes to specific items like bandwidth.

Seeding the clouds

We contacted each vendor, described our requirements and waited for the proposals.

Each vendor has a different process to arrive at a quote for the resources we asked for, which amounted to a small subset to the wide array of possible offerings in each vendor's menu. While each vendor had a different list of options, there were many commonalities. Ordering virtual private cloud or enterprise cloud services meant getting dedicated machines with gear we wanted and a connectivity method that would link our network operations center at n|Frame in Indianapolis to the vendor's resources through VPN connectivity, which should be used as a demarcation point for both security and cost purposes.

BlueLock's hardware choices were among the narrowest, but they won points for having a thorough and deliberate quotation and subsequent provisioning process. They use forms made of Excel worksheets to exchange information, but the interactivity of information exchanged was thorough and well thought-through. By contrast, Rackspace offered the most flexibility in many ways.

Terremark's rapid speed of delivery (three days) earned the product high marks as it delivered quickly and to spec -- all things we like in a cloud vendor. But the other vendors weren't far behind -- BlueLock delivered in five days and Rackspace in six.

BlueLock

BlueLock has an openly published security process, which initially intrigued us, and we were reminded of an almost military provisioning process. We e-mailed them with our desired configuration, and Bluelock responded with a detailed proposal. Bluelock creates the offering from a source document build list. Once we said "go", Bluelock created the entire private cloud, operating systems deployment, initial security, IP routing, and so on. We didn't create the virtual machines and BlueLock provisioned the VMware instances (VMware 3.5 at this writing; 4.0 soon). We received dedicated hardware running on HP blades, which are their only hardware platform.

For connectivity via VPN and firewalling, BlueLock provided a CheckPoint SSL VPN whose administrative interface doesn't work with very many browser platforms; we tried various setups but only were able to get it to work in Windows XP and Internet Explorer (and Firefox 3.5 with Java installed). Windows 7 with IE8 or Firefox, Mac OS X 10.5/10.6.x with Safari, Firefox, did not work at all. Once inside CheckPoint, it works well and it's an enterprise class workhorse firewall and VPN. Bluelock was also able to pass our not-a-Cisco VPN test, by connecting to our Vyatta router/VPN appliance quickly.

The management interface to our 10 operating systems instances could have been better. There is no Web interface for accessing VMs (you can only connect to instances directly after connected through the SSL VPN or through IPSec site-to-site VPN, we tried both). Cloud administration was stiff. Bluelocks's own Vital Signs portal is a Web-based shell program that in turn calls other administrative applications. Vital Signs displays choices including a Vital Signs Diagram (which wasn't useful, as it shows a user count, and our agreement did not concern users, so it displayed - one user), and Event Monitoring Portal (the FOSS tool, Nagios), a Trend Portal (the FOSS tool Cacti), a non-working Reports screen, a Ticket and Support System (trouble ticket submission and process control), a portal user account maintenance facility, and FAQs.

Nagios is an open source network monitoring tool that we used to monitor network services such as http or mysql servers, along with whether the host is alive (ping test). We could also set alarms or notifications if a Nagio-tested service failed. The Cacti trend portal showed us virtual machine and firewall information. Cacti does a great job of showing time series sample graphs of CPU usage, network activity, memory usage and disk usage. We found Bluelock's Vital Signs Ticket and Support System to be frustrating, as it gave us only summarized information and no transaction or billing history. The Vital Signs portal isn't well connected, in terms of applications integration, as pieces can't be related together as objects in easy ways. While most of the discrete applications are useful, they're very disjointed.

We logged on to check BlueLock's administrative interface, then dove into forming our test suite, which consisted of installing LAMP/WAMP onto each OS instance that had been created. We checked Bluelock's performance with an Apache benchmark. It turned out that all of the vendors performed within a narrow window.

We tested storage expansion, which was simply a matter of submitting a new support ticket. And Bluelock configured the IPSec tunnel correctly -- except for our public IP, none of the resources could be seen, and the CheckPoint firewall and tunnel manager kept it that way.

BlueLock had a very fast connection to our NOC -- uploads at 7.26Mbps and downloads at 8.8Mbps. But it's also located only a few miles away from our n|Frame NOC resources (our subscribed bandwidth was 1Mbps burstable to 10Mbps).

Overall, BlueLock's negotiation process is good, and its security components were well-managed. The BlueLock administrative method had applications that feel like separate products. Nothing is really connected together, most portals launch in another browser window, some even require a separate login/password combo. Administration is unnecessarily confusing using these tools. And since BlueLock controls changes to the operating systems deployed, the time between ticket submission and a change could be considerable. We wanted to occasionally use our root account just to get things done.

Terremark

Terremark's negotiation process is less formal than BlueLock's, although all of our private cloud metrics were met fully by Terremark. Terremark's hardware offerings are just slightly more expansive than those from BlueLock, as Terremark uses HP 580 and 585 servers. Terremark also offered us a variety of bundles that were pre-defined hardware/software asset combinations.

The build-time was shorter -- they were the first online and were ready-to-go quickly, although part of the speed came from the fact that Terremark didn't provision our instances of RedHat, and only offered Windows 2008 (not R2) server instances, with no maintenance, although it can be procured.

We told them the specs, they replied with a few questions, and in a couple of days, the components were built and we connected our NOC and the Terremark NOC. Terremark used virtual machines, like BlueLock, as the substrate for our requested network, and the connections to our Vyatta router/VPN appliance integrated quickly with their Cisco components.

Administrative interface

In the interest of time, Terremark had us provision our own virtual machines, which was a simple task. We were allocated the desired number of CPUs, RAM, disk and network for us to divide into the "shape" of the cloud we wanted. The Terremark-developed DigitalOps administrative app interface was used to deploy our Windows and Linux instances from one-click templates. Terremark supplied the Windows licenses (ostensibly from a volume license) and supplied Red Hat operating systems -- but we registered licenses supplied to us by Red Hat. Rollout, therefore, was drama-free and just 10 clicks for 10 instances. Terremark can optionally install everything for you at additional cost. We had the option of rolling out other types of server licenses operating systems from ISO images as well.

DigitalOps has a user interface that's separated into two main tabs, Environment and My Account. Under Environment there are three tabs: Resources, Devices and Network. The Resources tab displays information about processor, memory and storage usage. The main Resources page has a summary of each for the past 24 hours and is very easy to understand. We could get more detailed information by using the sub-tabs about each individual component (processor, memory, storage) if desired. The Devices tab lists all the virtual machines that we created, and the virtual machines can be sorted into groups and rows. We could create virtual machines from pre-built templates or create a blank server using our own ISO, as mentioned.

We could also use a VPN Connect button that allowed us to link to an SSL VPN (which is required to actually connect to the consoles of the virtual machines created). The final tab in the Environment section is network. Here we could view the IP networks assigned to us, internal, external and public IP addresses. We could also setup firewall and port-forwarding rules, although, they are very basic and we couldn't customize it very much.

Site to Site VPNs were a separate package deal, but possible to do using the IPSec protocol. Terremark only supports certain hardware or software VPNs, but they will do a "best effort" to try to get things working, if you have something different. We had something different, the aforementioned Vyatta appliance and we got the VPN working with minimal trouble. Once everything was setup, we ran some brief upload tests between our NOC and their servers. During an ISO transfer using scp, we maxed out around 120KBps (average). Normal FTP was about the same around 125KBps. The connection was limited to 1Mbit (not burstable) which is about 128KB, so it was pretty much maxing out the connection.

Terremark supplied an older VMware console plugin (which oddly doesn't work in Windows 7 under IE 8 or Firefox 3.6 but did work in Firefox 3.5.7) but none of the other competitors offered any option to connect to the virtual machines via their respective Web interfaces -- and Terremark did. This wasn't as much of an issue with the Windows virtual machines (meaning console virtual machine access) as the Windows Server virtual machines had Remote Desktop turned on to give us access. We had a few small quibbles with the templates used to generate the RHEL virtual machines, as the template did not create a user besides root (therefore, we couldn't SSH in, as root SSH is disabled by default).

The Terremark committed bandwidth pricing is complicated and is based on a "95th percentile" scheme, where they take the top 5% of your traffic for the month, drop that from calculations and use the final 95% of the bandwidth you used to figure out a price. You must purchase a Committed Bandwidth package. Ours was the 5Mbit package, which is $25 per Mbit, so $125 in total.