Facebook rolls out storage system to wrangle massive photo stores
Homegrown system, Facebook Haystack, built to handle multiplying files of user photos
Computerworld - Needing to better deal with 50 billion files worth of photos, engineers at Facebook are installing a new photo storage system they say is 50% faster than traditional systems.
The storage system, dubbed Haystack, has been under development in-house for the past couple of years, and Facebook has been rolling it out in limited test versions to parts of the network for the past few months. The company expects to use Haystack to store all Facebook photos by next week, according to Bobby Johnson, director of engineering at Facebook.
And Jonathan Heiliger, vice president of technical operations, told Computerworld today that based on tests, Haystack is more than 50% faster than traditional photo storage systems.
"In terms of cost, if it's twice as efficient, we can have 50% less hardware," said Johnson. "With 50 billion files on disk, the cost adds up. It's essentially giving us some [financial] headroom."
Johnson and Heiliger said they began building the new storage system to better handle the growing number of photos Facebook has to store. Many of their 175 million and 200 million users share photos of everything from their pets to vacations, weddings and days at the beach. That means users are posting and calling up their own photos, as well as their friends' and family members' photos. Keeping the system running efficiently was a growing challenge.
Johnson noted that Facebook deals with 15 billion photos - not including all of the replications. User data grows by 500GB per day. And Facebook has 50 million requests per second to its back-end servers.
A spokesman for Facebook said more specifics about the new system will be released in a few weeks.
Johnson, though, said the system is so much faster than the previous one because of changes made to its setup. Haystack is tailored for small files that don't change very often, instead of for a small number of large files that are changing all the time. Traditional file directories also need file names, and a lot of resource cost goes to just finding the files. The new system uses ID numbers instead of names; that mapping is very small and doesn't involve directory structures or file names.
Johnson said that so far, the rollout of the new system has gone very smoothly.
Facebook, once regarded as the up-and-coming social network, had almost 222 million unique visitors last month, while MySpace came in at 125 million, according to online researcher comScore Inc. That's a dramatic change, since the Facebook-MySpace race for unique visitors was a near dead heat in April 2008.
The company is closing in on a big milestone -- 200 million users, executives said today.
Read more about Web Apps in Computerworld's Web Apps Topic Center.
- Best iPhone, iPad Business Apps for 2014
- 14 Tech Conventions You Should Attend in 2014
- 10 Desktop Apps to Power Your Windows PC
- How to Add New Job Skills Without Going Back to School
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- Face Time Anytime Real-time communications facilitates team collaboration from nearly anywhere in the world. With facts and figures you can use to justify an investment
- Riverbed Stingray Application Firewall: Securing Cloud Applications with a Distributed Web Application Firewall Responsibility over IT security is moving away from the network and IT infrastructure and to the application and software architecture itself. IT organizations...
- Now is the time to implement a video conference solution Video conferencing is getting a lot of buzz lately due to the recent cost decrease, making it tangible for many law firms. It's...
- Video drives engagement Achieving maximum results means building a solid platform and network infrastructure. As digital age unfolds, it's clear that the ability to communicate effectively...
- Webinar: Building a Big Data solution that's production-ready Big data solutions are no longer just a nice-to-have.
- Meg Whitman presents Unlocking IT with Big Data During this Web Event you will hear Meg Whitman, President and CEO, HP discuss HAVEn - the #1 Big Data platform, as well... All Web Apps White Papers | Webcasts