Digg Dips Deep Into Open Source

Computerworld - SANTA CLARA, Calif. -- Information technology staffers at Digg Inc. credit two particular features of the companys LAMP-based server cluster for helping its Digg.com news aggregation Web site maintain speedy performance in the face of rapid usage increases.

The site, which lets visitors vote on  or digg  their favorite news stories hosted on other sites, recently passed the 1.2 million user mark, according to Elliot White III, an engineer at Digg who spoke at MySQLs user conference here last week.

Digg has about 100 servers that run a combination of Linux, the Apache Web server, the MySQL database and the PHP scripting language  all open-source technologies that are collectively referred to as LAMP. The systems, which are scattered in multiple data centers, include about 20 database servers, 30 Web servers and a few search servers running the open-source Lucene search engine. The rest of the systems operate as backup machines.

In Diggs architecture, a load balancer sends queries to PHP servers, MySQL slave servers that feed data to the PHP servers, and a MySQL master server that feeds data to the slaves. Thats a fairly standard setup. But White said that to get away from sending raw queries against the database, the San Francisco-based company uses open-source memory caching software called Memcached.

First developed for use by LiveJournal Inc.s online journaling Web site, Memcached stores chunks of data that can be pulled out and used to dynamically create a Web page. Conventional caching technologies, which store entire Web pages, would be too slow and inefficient for a site that changes continuously like Digg.-com, White said.

The other atypical feature of Diggs setup is its use of what engineer Tim Ellis called sharding  a term apparently coined by developers at Google Inc. Sharding involves breaking a database into smaller parts to improve performance by isolating heavy workloads.

If 90% of your data is within a certain range and you can get that part working really fast, you can help customers, Ellis said. Then its OK if the remaining 10% is slower.

A database can be sharded by table, date or range. The process is similar to partitioning but with some key differences, Ellis said. For example, sharding usually involves divvying up data onto different physical machines, but partitioning is typically done on the same piece of hardware.

Breaking a database into several smaller pieces can mean more work because of the inability to use common SQL commands, such as table joins, Ellis noted. Developers dont like this crazy stuff, he said.

Digg is really lucky in that 98% of the time, users are reading data rather than writing it to the server, Ellis noted. Most people come to Diggs front page, read it and leave, which is kind of nice, he said, drawing laughs from the audience.