Facebook stays with its homegrown infrastructure

IDG News Service - Facebook regularly reviews whether to continue using its own data centers or hand off its processing to a cloud service provider, and its operations remain on homegrown infrastructure.

"We look at, probably once or twice a year, does it make sense for us to build more infrastructure in the cloud, and on someone else's backbone, and less for ourselves?" said Jonathan Heiliger, vice president of technical operations at Facebook, in an onstage conversation at the Structure 2010 conference in San Francisco. "We've determined, for two reasons, that it doesn't, each time we've done this analysis," Heiliger said.

First, the cost to Facebook of building and maintaining its own infrastructure is still lower than the bids it is receiving from cloud service providers, he said. Second, the homegrown infrastructure gives Facebook more freedom.

"It gives us a tone of flexibility. We don't have to wait for someone to go develop a feature. We just roll it out ourselves," Heiliger said. By using its own infrastructure, Facebook can put in a different CPU architecture, add a new networking device, or introduce traffic acceleration when and where it likes. The company is also free to shift traffic around and place caches where it makes the most sense, he added.

Several other large-scale Web sites, with tens of millions of users, also have been investing in their own infrastructure recently to cut costs or boost performance, Heiliger said. He advised a growing Web company to compare the total cost of ownership of cloud versus homegrown infrastructure and the value of having internal control over it.

One other factor to consider is that, typically, the amount of reimbursement available under most cloud service-level agreements is capped at the value of the contract rather than the value to the company or its customers of the lost operating time, Heiliger said. The latter may be much higher.

Facebook's is no ordinary workload. The social-networking hub, which has more than 400 million members, needs to carry out 50 million or more operations per second and handle more than 1 million photos per second, Heiliger said. He expects those numbers to multiply by 10 over the next year.

At last year's Structure, Heiliger said Intel and AMD's latest CPUs weren't delivering the performance gains the companies had promised. His company uses both. Facebook has been working with software and kernel developers from the chip companies over the past year, and the chips' performance has improved as a result, he said in an interview after the session. Heiliger noted that Intel's and AMD's chips aren't inadequate to Facebook's needs but only fell short of what had been promised.

However, "We'd always like more," he added.