Open-source tools keep Twitter up and running

To curb disruptions and scale up its service while keeping costs down, Twitter has drastically changed its core infrastructure and has adopted some open-source tools.

Twitter processes about 6,000 messages per second -- that's more than 500 million a day and about 3.5 billion a week. The company set a record earlier this year, with 143,000 messages per second during the airing of a movie in Japan, said Chris Aniszczyk, Twitter's head of open-source computing, at a recent conference in Europe.

When Twitter debuted in 2006, it used a monolithic Ruby on Rails application. That worked fine until 2008, when the microblogging service started to suffer a lot of "fail whales" -- the company's term for service disruptions.

The company decided to break the one application it had been using for everything into different services, Aniszczyk said. It also started using open-source tools such as Apache Mesos, a cluster manager; Netty, for creating high-performing protocol servers; and Scalding, for writing big data jobs.

And in another move, Twitter switched its core infrastructure to a Java virtual machine. The end result, said Aniszczyk, is a performance improvement and fewer disruptions.

This version of this story was originally published in Computerworld's print edition. It was adapted from an article that appeared earlier on Computerworld.com.

To express your thoughts on Computerworld content, visit Computerworld's Facebook page, LinkedIn page and Twitter stream.
Windows 10 annoyances and solutions
Shop Tech Products at Amazon
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.