Library of Congress saves 500 million tweets per day in archives
The government library is storing the messages as part of its efforts to archive all Twitter activity
IDG News Service - The Library of Congress is now storing 500 million tweets per day as part of its efforts to build a Twitter archive, and has added a total of about 170 billion tweets to its collection.
Twitter signed an agreement in April 2010 to provide the library with an archive of every public tweet since the company went live in 2006, and the Library recently provided an update on its progress. The initial stage of the project, which includes a complete copy of all tweets covering that four-year span, will be finished by the end of the month.
The project is now shifting its focus to making the collection accessible to lawmakers and researchers. The library has permission from Twitter to share the tweets with vetted researchers at least six months after they were published, provided they are not used for profit or redistributed.
"It is clear that technology to allow for scholarship access to large data sets is lagging behind technology for creating and distributing such data," the library noted in a public document about its progress.
The library said the initial four-year archive contained about 21 billion tweets that take up 20 terabytes when uncompressed, including data fields. It continues to receive and process messages from Twitter, which are now organized into files representing hour-long segments by Gnip, a service provider with which Twitter also partnered in 2010 to provide commercial access to the full range of tweets.
The full archive now requires 133.2 terabytes for two compressed copies, which are stored on tape in separate locations for safe keeping.
The library said it has already received 400 inquires from researchers for access to the archives, for research topics including citizen journalism, vaccination rates and stock market trends.
The library said it already maintains similar collections, such as an archive of web sites related to government and policy matters that is over 300 terabytes in size.
The government library's collection includes over 34.5 million books and 66.6 million manuscripts. It is officially the working library of the U.S. Congress, but also serves as a national archive of written works for the country.
- 20 Best iPhone/iPad Games of 2013
- Google Chromebook Buyer's Guide
- 10 Signs You're Probably a Techie
- 8 Things Kindle Fire HDX Does That iPad Air Can't
- Slideshow: 5 ways to lock down your mobile device
- Slideshow: 10 mistakes companies make after a data breach
- How to rob a bank: A social engineering walk through
- Which smartphone is the most secure?
Cloud has proven to be a sticky marketing term, even (especially?) among non-technical consumers. There is a growing consensus that “the cloud” promises a quicker and cheaper way of purchasing IT than the traditional, build-it-yourself model. This in turn has led to internal debates between application developers and infrastructure teams on the best approach for implementing “the cloud” within IT organizations. (And of course, IT architects know that when they hear management say, “We want to move to the cloud,” they will be the ones who have to figure out how to do it!)
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Bring Networks and Applications Closer--Cisco ONE
- A series of sweeping trends is placing new requirements on the tried-and-true network model--requiring network infrastructure and applications to communicate. Get the open...
- Lippis Research Reviews the Cisco Catalyst 2960-X
- In this Lippis Report Research Note, Lippis Research reviews the latest edition of the "most popular access switch on the planet" -- the...
- Design Guide--Scaling Up to a Campus-Wide LAN
- Is it time to scale your network environment to a campus wired LAN? Here's the framework you need to set up your LAN...
- Comprehensive Security: Cisco Catalyst 2960 Series
- With a rich and comprehensive set of security features, Cisco Catalyst 2960-X and 2960-XR Series Switches can help you address networking megatrends such...
- Be Energy Efficient--The Cisco Catalyst 2960 Series
- How much energy could be saved if all 230 million Layer 2 and 3 fixed managed switch ports sold in 2012 were as... All Government IT White Papers
- Tame the data explosion with IBM Real-time Compression Learn about IBM's unique Real-time Compression technology that can enable storing up to 5x as much data in the same storage space.
- IT Transformation: Making the Move to Services IT transformation is happening in businesses of all sizes
- How SIEM Addresses the Challenges of Big Security Data This webcast will help you understand today's big data security challenges and how intelligent and scalable SIEM solutions give IT the tools and...
- BlackBerry Enterprise Service 10 In Action Designed for business leaders, IT decision-makers and IT administrators, this webcast features organizations that have migrated to BlackBerry® Enterprise Service 10 - or...
- BlackBerry Enterprise Service 10 Implementation Solutions - BlackBerry Technical Services to Make your Deployment Fast, Effective and Robust Find out how BlackBerry® implementation and migration services can help you optimize your Enterprise Mobility Management infrastructure and ease your transition to BlackBerry®...
- All Government IT Webcasts