Open-source tools keep Twitter up and running

To curb disruptions and scale up its service while keeping costs down, Twitter has drastically changed its core infrastructure and has adopted some open-source tools.

Twitter processes about 6,000 messages per second -- that's more than 500 million a day and about 3.5 billion a week. The company set a record earlier this year, with 143,000 messages per second during the airing of a movie in Japan, said Chris Aniszczyk, Twitter's head of open-source computing, at a recent conference in Europe.

When Twitter debuted in 2006, it used a monolithic Ruby on Rails application. That worked fine until 2008, when the microblogging service started to suffer a lot of "fail whales" -- the company's term for service disruptions.

The company decided to break the one application it had been using for everything into different services, Aniszczyk said. It also started using open-source tools such as Apache Mesos, a cluster manager; Netty, for creating high-performing protocol servers; and Scalding, for writing big data jobs.

And in another move, Twitter switched its core infrastructure to a Java virtual machine. The end result, said Aniszczyk, is a performance improvement and fewer disruptions.

This version of this story was originally published in Computerworld's print edition. It was adapted from an article that appeared earlier on Computerworld.com.

FREE Computerworld Insider Guide: Five IT certifications that won’t break you
Join the discussion
Be the first to comment on this article. Our Commenting Policies