IDG News Service - Eschewing popular choices such as XML, CSV and JSON, Twitter has opted to format the back-end storage of its user and systems data with a relatively unknown format pioneered by Google, called Protocol Buffers.
With the company storing 12TB of this data each day for later use, the decision of which format to use was a crucial one.
"Getting your data formats right is everything," said Twitter analytics lead Kevin Weil, during a talk at the HadoopWorld conference in New York on Tuesday.
The company is planning for the time when it will have to house "a trillion Tweets," Weil said, and it wants tools in place to analyze this information. The combination of Protocol Buffers, along with Hadoop and other associated technologies, should streamline this job, Weil said.
When stored, each short message, or "tweet," consists of 17 fields, six of which have at least one subfield, he explained. And the company will probably add more fields to these schema in the years to come.
In addition to the tweets the company's users supply, Twitter keeps internal log data on more than 80 different types of operations that occur within its systems, Weil said. Much of this log data is aggregated by Facebook's open-source technology Scribe.
The choice of a format to store all this data was a difficult one. One obvious choice is XML (Extensible Markup Language), but that protocol is "very wordy," Weil said, referring to how the name of the tag accompanies each data element.
Under XML, "one petabyte for a trillion Tweets might become 10 petabytes for a trillion Tweets," he said.
At the other end of the spectrum is CSV (Comma Separated Values). As the name suggests, CSV separates each data element only with a comma. While simple, it is not good for nesting data elements in subfields, Weil explained. Also, if the schema is changed, the resulting programming it would take to accommodate data in the old schema would be considerable.
A downside to all of these protocols is that, in order to get the data in and out of applications, developers have to repeatedly create data structures to encode and parse the data, work Weil considers "rote."
Protocol Buffers, used widely within Google, is an extensible protocol for serializing data, one Google claims is simpler than XML. And it can automate the process of recreating the data structures within applications.
"You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages," a Google tutorial on Protocol Buffers states. "You can even update your data structure without breaking deployed programs that are compiled against the 'old' format."
- Silicon Valley's 19 Coolest Places to Work
- Is Windows 8 Development Worth the Trouble?
- 8 Books Every IT Leader Should Read This Year
- 10 Hot Hadoop Startups to Watch
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- Windows® XP Migration: Protect and Secure Critical Data With the end of the Microsoft Windows XP operating system's lifecycle on April 8, 2014, businesses are faced with the decision to migrate...
- Exponentially Accelerate Data Protection and Recovery with Simpana 10 IntelliSnap® Snapshot Management Technology Are you making the best use of your storage array snapshot functionality? CommVault Simpana 10 IntelliSnap technology manages hardware-based snapshots across multiple vendor...
- Simpana IntelliSnap Technology Datasheet With IntelliSnap you can maximize the value of your snapshot technology while dramatically reducing management overhead and complexity.
- Enhancing Application Protection and Recovery with a Modern Approach to Snapshot Management This CommVault Business Value and Technology White Paper explains how Simpana IntelliSnap® Recovery Manager can make your application recovery fast and reliable.
- Live Webcast LIVE EVENT: 5/7, The End of Data Protection As We Know It. Introducing a Next Generation Data Protection Architecture. Traditional backup is going away, but where does this leave end-users?
- LIVE EVENT: 5/7, The End of Data Protection As We Know It. Introducing a Next Generation Data Protection Architecture. Traditional backup is going away, but where does this leave end-users?
- Make or Break: New Auto Products Must Go To Market On Time This Webcast quantifies the value of time to market for the auto industry and highlights how Primavera Enterprise Portfolio Management can help organizations. All Data Storage White Papers | Webcasts