After bumpy start, transit agency finds virtualization a smooth ride

Suburban sprawl might make a great business case for a transit agency, but when it came to servers, Canada's Societe de transport de Montreal (STM) drew the line.

Mike Stefanakis, senior systems engineer at STM, says that the main reason he started looking at virtualization technology was to prevent server sprawl. He wanted consolidation, particularly for development servers at the agency, which provides over 360 million bus and metro rides each year.

"We crunched the numbers and realized that our growth was going to cause a few problems in the near future," he says. If things kept going as they had, the agency would need an additional 20 to 30 servers each year, on top of its existing base of 180 primarily Wintel machines. "Too many servers were going to be needed to feed the needs of our users and clients," Stefanakis says.

Looking out five or six years, STM would need twice the floor space, plus lots more electricity and air conditioning, to keep the shop running smoothly.

But even though staffers were convinced of virtualization's benefits pretty early on, the agency's end users didn't necessarily feel the same way.

When Stefanakis began virtualizing the first test applications back in 2005, there was some hesitation on the part of users to try the technology, he says. Several factors contributed to the initial resistance. For starters, there was a fear of the unknown. There were questions like "How stable is this new technology?" and "What do you mean I will be sharing my resources with other servers?" Potential users thought the new technology might slow them down.

Getting buy-in

To help users get over their fears, Stefanakis focused on giving people the information they needed, while explaining the advantages of the new technology. Among them: great response time for business applications and baked-in disaster recovery. If anything does fail, restoration is just a quickly restored image away.

Stefanakis and his staff kept "talking up" the technology and its benefits. "Virtualization came up in every budget, strategy and development meeting we had," he recalls. "We made sure the information was conveyed to the proper people so that everyone in our department knew that virtualization was coming."

STM has been staging production servers in its virtual environment since December 2005. The first virtual machine was staged in STM's testing center as a means of quickly recovering a downed production server, says Stefanakis.

Once the first few applications were implemented, user resistance quickly became history. "After people see the advantages, stability and performance available to them on a virtual platform, they tend to lose any inhibitions they previously may have had. The psychological barrier for virtualization has been broken," Stefanakis says.

Patrick Hardy, STM's senior network architect, has been using virtualized applications for over a year. These include Cisco Secure Access Control Server (ACS) Version 4, Citrix Presentation Servers, MS-AD controllers, DNS and DHCP servers, among others.

A happy camper

Hardy says he has noticed several improvements with the virtualized environment, including increased availability -- both uptime and response time. "We now have near zero downtime for system maintenance," he says. When system failures do occur, he explains that virtual servers are "shifted" live from one physical box to another, "without losing a single ping."

The benefit, Hardy says, is that recovery and maintenance of servers becomes seamless and transparent to end users. He says that there have not been any performance issues.

"Virtualization is the buzz amongst all my colleagues," he says. For instance, standard Microsoft 2003 virtual servers can be provisioned within 15 minutes by server administrators. This is critical, he explains, when application administrators need to test or work out a bug on a production server. For the network administrator, this means fewer network ports and less money needed to provision. Trunks can carry multiple virtual LANs (VLAN) to a single box, containing many virtual servers in many different subnets. "We hope to see in the near future" support for 10 Gigabit Ethernet network interface cards for host computers in STM's data centers, he says.

STM has lots of company in its virtualization efforts. A Forrester Research Inc. report from analysts Frank E. Gillett and Galen Schreck found that globally, 75% of the 1,221 companies they surveyed are aware of server virtualization technologies, 26% have already implemented it and, as of last summer, 8% more had piloted it. The reason, according to the report, is that virtualization's cost-benefit ratio is compelling, including the ability "to reduce costs while making their infrastructures more flexible over time."

Still, not everyone experiences flawless implementations. In our related story, one analyst talks about some of the most common problems, including cost accounting, politics and lack of multiplatform management tools.

Moving to a virtual infrastructure

Gordon Haff, an analyst at Illuminata Inc., a Nashua, N.H.-based consultancy, says that there is now a push going on past plain-vanilla server virtualization to what is sometimes referred to as "virtual infrastructure."

"The idea is that the abstraction layer provided by virtualization makes it much easier to move and manage workloads by breaking their ties to specific physical servers," Haff says. "There is certainly this broad vision of the future -- with everything virtualized and dynamic. You don't see that in many places yet. However, you do see a significant number of more modest virtual environments."

Indeed, many organizations are just getting ramped up with virtualization. That certainly is still true for STM, which has eased into the technology. It was a daunting prospect, especially when Stefanakis considered putting a lot of eggs into one new technology basket. He says that with more conventional strategies, if STM lost one physical server, it would lose one workload or application. With virtual servers, if STM loses one physical server, there is the potential to lose between 15 and 30 workloads.

The biggest challenge was coming up with an architecture and strategies that would make our farm as secure and redundant as possible," he explains. "We had to maintain a certain level of performance and at the same time stay within the budget that was allocated for the project. All of this had to be done right out of the starting gate. We couldn't afford growing pains because any downtime or performance lags would have been interpreted as instability in our new platforms and most likely killed all confidence in it."

STM's architecture

STM is mainly a Microsoft/Intel shop, with around 190 production and development servers running Windows 2000 and Windows 2003, and a smattering of IBM/AIX servers. The AIX boxes are also being virtualized, but that's being handled by another project team at the agency.

A mix of applications on the Wintel machines includes SQL, Oracle, IIS, SAP, Exchange/Outlook and MS Office.

"Virtualization has given us the advantage of having a flexible environment," Stefanakis says. Since servers' loads are no longer attached to the physical platform, he says that this offers many advantages in terms of maintenance, upgrades and hardware replacement.

For example, in 2006, STM's major focus was to replace 30 aging IBM Netfinity servers at a cost of approximately $212,000. This would have been for just one-to-one replacement of the physical machines, without any frills like redundancy, he explains. In comparison, the VMware software and associated hardware cost $178,000 but had enough capacity to house 45 servers instead of 30. So even with a conservative ratio of 15:1 for the server consolidation using high-end IBM 3950 servers, STM was able to improve performance and provide physical redundancy for all servers.

Even with the consolidation of the initial 30 machines, there's room enough to add another 10 to 20 servers into the mix.

Although the initial plan was to implement the original 30 server loads onto three of the new IBM servers, Stefanakis' team actually wound up with eight new IBM 3950 units. "From the time we sized the solution, to the actual purchase approximately nine months later, we found more money from various projects to purchase an additional five servers," he says. There are two IBM 445 machines running VMware as well.

Of the 10 machines, Stefanakis says that there are five in full production and another five in pre-production. Some 79 virtual servers are running on the five production machines.

The virtual application mix

STM is virtualizing a variety of applications, including some with Oracle back ends, where the database servers are also virtualized; Crystal reports; and SQL applications. In addition, Citrix servers are virtualized, along with infrastructure servers such as proxy, DNS, domain controllers, FTP servers and IIS servers.

Although Stefanakis researched other vendors, including Microsoft, before moving ahead with VMware, he believed that VMware offered the best overall features. "They also enjoy the best reputation in the market, and have been doing virtualization longer than anyone else," he says. Two of the features he really likes are VirtualCenter, with its centralized management, automation and server optimization, and VMotion, which allows live migration of virtual machines without service interruption.

STM's goal is to continue to convert as many physical servers as possible to virtual ones. "We usually classify the life cycle of a server as five years," Stefanakis notes." To justify the replacement of physical servers with virtual ones -- except when special circumstances come into play -- on a budgetary level, I would have to say that all physical servers that are candidates for virtualization should be completed in the next three to four years."

STM's goal this year is to transform an additional 40 physical machines to virtual, and stage an additional 15 to 20 virtual servers.

Disaster recovery becoming popular justification

Looking ahead, Stefanakis sees VMware as being the main platform on which STM's server disaster recovery strategies will be based.

Andi Mann, an analyst at Enterprise Management Associates, an IT consultancy in Boulder, Colo., says that both disaster recovery and business continuity planning are popular reasons for adopting virtualization technology, along with the desire for increased flexibility and agility, and reduced downtime.

"Server consolidation is popular, but it is a one-time deal, and does not provide ongoing value to the business," says Mann. "It is a short-sighted IT cost benefit -- which is not to be sneezed at, but not strategically valuable, either. Increasingly, IT is realizing how virtualization can deliver rapid and dynamic business agility, which can be a competitive differentiator. This is the long-term trend for virtualization -- delivering extremely agile IT to support rapidly changing business needs."

Sometimes, though, things can get a bit too agile. "Because we can now provision a server within minutes instead of weeks, we noticed a danger of encouraging virtual server sprawl," Stefanakis says.

And yes, he confesses, showing off to colleagues does have something to do with it.

A while back, a developer came to ask him for a virtual server. "I allocated it to him while we were talking about [my] wife and kids," Stefanakis recalls. "When he got up to leave, he asked me when he could expect his server, and was told that it was ready. We had a bit of fun with it, until the novelty wore off and we realized that this can get out of control. So, unfortunately, we became serious again."

To avoid this type of off-the-cuff virtualization, Stefanakis suggests that staffers adhere to very strict controls for provisioning virtual servers. But it's not always easy to take his own advice, particularly when once-reluctant end users start ordering up more virtualized applications.

"The psychological barrier was broken," Stefanakis says, "and now users will ask for a new server as if they are ordering a coffee and danish."

Mary Ryan Garcia is a Coram, N.Y.-based freelance technology journalist specializing in enterprise-wide virtualization, storage and security issues. She can be reached at

Copyright © 2007 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon