Government open data proves a treasure trove for savvy businesses
A team of nearly 15 data analysts processes more than 60,000 retirement plans per year while integrating data captured from SEC filings, financial industry regulatory authorities, census data and public websites. Once collected, the data is then cleansed and linked together using proprietary mapping algorithms.
"There's a big difference between free open data and actionable intelligence," says Alfred. "Companies don't want to purchase data that requires data analysts and infrastructure and integration. So there's a ton of work on both the manual side and the engineering side to get the data into a format that can be reliably used by our clients." That's a complex process that, he says, could be simplified with a bit of help from the government.
The good news is that the government is taking steps to improve data quality. Calcbench has benefited from that effort. The New York-based startup has created a sophisticated engine that turns complex financial data such as earnings statements, cash flow statements and companies' balance sheets into a more readable format. To ensure accuracy, Calcbench uses proprietary artificial intelligence tools that sift through the data to detect errors such as misdated year-end reports. Finance professionals such as investors, auditors and industry researchers use the resulting repackaged and cleansed data to compare financial ratios, examine entire industries and review competitors' disclosure data.
"A lot of what we're charging for isn't the data," says Alex Rapp, Calcbench's CTO and co-founder. "It's how we structure it, how we store it for you and how we solve a tremendous number of business problems by making data comparable across different companies and industries. That's something the data doesn't do itself."
Nor is it a service that would have been possible without the U.S. government's insistence, starting in 2009, that the financial information in corporate SEC filings must be in XBRL (Extensible Business Reporting Language) -- a freely available and global standard format for exchanging business information. Whether you're downloading quarterly filings directly into spreadsheets or analyzing year-end statements using off-the-shelf software, XBRL converts fragmented financial data into a machine-readable format. And that makes life easier for Calcbench's techies.
But as government agencies inch toward cleaner, more structured data sets, a handful of startups are offering powerful new tools for dealing with older data. "These intermediaries are going to make open data more accessible to other types of companies because the reality is, open data is still pretty messy at the local, state and federal level," says Gurin.
One of those startups is New York-based Enigma Technologies, which has created a platform that allows users to access open data via an API or a Web dashboard. The company uses Web crawlers that gather public data from sources such as dot-gov websites and FCC documents regardless of whether the information is stored in a Zip file, an Excel spreadsheet or a multigigabyte database. Next, the platform creates a relational infrastructure that centralizes all of the open data and creates links between matching data sets. A single interface offers users access to more than 100,000 sources of finely curated data. And because Enigma's solution is offered as an API, developers can borrow the technology to build dashboards that serve their industry-specific data retrieval needs.
"We're able to provide access to this universe of public data that is otherwise totally obscure and in the shadow," says Enigma co-founder Marc DaCosta.
In fact, there are plenty of ways for IT departments to tap the business value of open data. Some will opt to use tools from companies like Enigma. Others will staff their IT departments with Ph.D.s, develop sophisticated algorithms and build robust storage networks. And some will wait until government protocols catch up with corporate demands. Whatever the game plan, says Gurin: "Open data may not be the way for everybody, but it is the way forward-thinking CIOs are thinking about business."
Waxer is a Toronto-based freelance journalist. She has written articles for various publications and news sites, including The Economist, MIT Technology Review and CNNMoney.com.
Read more about Business Intelligence/Analytics in Computerworld's Business Intelligence/Analytics Topic Center.
- 2014 Gartner Magic Quadrant Report For the 7th year in a row, Riverbed is in the "Leaders" Quadrant of the 2014 Magic Quadrant for WAN Optimization Controllers. In...
- Eliminating Jenkins Sprawl: Going from CI to CD with Electric Cloud Many organizations have begun adopting Continuous Delivery (CD) practices to increase velocity beyond the build and test phase. This paper reveals how organizations...
- Dynamic Workload Automation Optimizes End-to-End Business Service Performance This white paper, sponsored by CA Technologies, discusses the market and technology forces that are reshaping today's enterprise workload management automation requirements and...
- Dell Software Foglight APM 5.9: Introducing Big Data Repository/Reporting Supporting Web Analytics and User-Centric APM Learn the capabilities of Dell™ Foglight™ Application Performance Monitoring (APM) 5.9, and how the deep visibility into transaction flows and User Experience capitalize...
- It's not too late...Get Your Mobile Questions Answered Live! How can IT provide seamless and secure mobile communications and collaboration for all? Join this live Webcast as IDG asks an expert panel...
- Capturing Data in Motion: Delivering Real-Time Insight from Data Streams This webcast will help organizations of all types and sizes learn about a technology and business strategy for tapping into the wealth of... All Business Intelligence/Analytics White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!