Government open data proves a treasure trove for savvy businesses
A team of nearly 15 data analysts processes more than 60,000 retirement plans per year while integrating data captured from SEC filings, financial industry regulatory authorities, census data and public websites. Once collected, the data is then cleansed and linked together using proprietary mapping algorithms.
"There's a big difference between free open data and actionable intelligence," says Alfred. "Companies don't want to purchase data that requires data analysts and infrastructure and integration. So there's a ton of work on both the manual side and the engineering side to get the data into a format that can be reliably used by our clients." That's a complex process that, he says, could be simplified with a bit of help from the government.
The good news is that the government is taking steps to improve data quality. Calcbench has benefited from that effort. The New York-based startup has created a sophisticated engine that turns complex financial data such as earnings statements, cash flow statements and companies' balance sheets into a more readable format. To ensure accuracy, Calcbench uses proprietary artificial intelligence tools that sift through the data to detect errors such as misdated year-end reports. Finance professionals such as investors, auditors and industry researchers use the resulting repackaged and cleansed data to compare financial ratios, examine entire industries and review competitors' disclosure data.
"A lot of what we're charging for isn't the data," says Alex Rapp, Calcbench's CTO and co-founder. "It's how we structure it, how we store it for you and how we solve a tremendous number of business problems by making data comparable across different companies and industries. That's something the data doesn't do itself."
Nor is it a service that would have been possible without the U.S. government's insistence, starting in 2009, that the financial information in corporate SEC filings must be in XBRL (Extensible Business Reporting Language) -- a freely available and global standard format for exchanging business information. Whether you're downloading quarterly filings directly into spreadsheets or analyzing year-end statements using off-the-shelf software, XBRL converts fragmented financial data into a machine-readable format. And that makes life easier for Calcbench's techies.
But as government agencies inch toward cleaner, more structured data sets, a handful of startups are offering powerful new tools for dealing with older data. "These intermediaries are going to make open data more accessible to other types of companies because the reality is, open data is still pretty messy at the local, state and federal level," says Gurin.
One of those startups is New York-based Enigma Technologies, which has created a platform that allows users to access open data via an API or a Web dashboard. The company uses Web crawlers that gather public data from sources such as dot-gov websites and FCC documents regardless of whether the information is stored in a Zip file, an Excel spreadsheet or a multigigabyte database. Next, the platform creates a relational infrastructure that centralizes all of the open data and creates links between matching data sets. A single interface offers users access to more than 100,000 sources of finely curated data. And because Enigma's solution is offered as an API, developers can borrow the technology to build dashboards that serve their industry-specific data retrieval needs.
"We're able to provide access to this universe of public data that is otherwise totally obscure and in the shadow," says Enigma co-founder Marc DaCosta.
In fact, there are plenty of ways for IT departments to tap the business value of open data. Some will opt to use tools from companies like Enigma. Others will staff their IT departments with Ph.D.s, develop sophisticated algorithms and build robust storage networks. And some will wait until government protocols catch up with corporate demands. Whatever the game plan, says Gurin: "Open data may not be the way for everybody, but it is the way forward-thinking CIOs are thinking about business."
Waxer is a Toronto-based freelance journalist. She has written articles for various publications and news sites, including The Economist, MIT Technology Review and CNNMoney.com.
Read more about Business Intelligence/Analytics in Computerworld's Business Intelligence/Analytics Topic Center.
- Reducing Expenses Starts with the Programming Skills of Operations Groups Reducing expenses can start with the programming skills of operations groups. Read this article to learn how to improve the bottom line, and...
- Big Data, Big Mess: Sound Risk Intelligence Through Complete Context This paper examines the insecurity of the small businesses in the supply chain and offers tips to close those backdoors into the enterprise.
- CIOs strive to harness Big Data while keeping an eye on the bottom line Read this whitepaper to learn how Red Hat Storage Server allows CIOs to confidently support business growth, manage cost and risk, capitalize on...
- Enterprise architects challenged to manage data explosion Read this whitepaper to find out how Red Hat Storage Server can allow enterprises to quickly and confidently deliver business applications that minimize...
- Live Webcast Charting Your Analytical Future - "Making predictive analytics part of your business processes" Webinar This session will show how predictive analytics can be used throughout the organization by anyone looking for answers and how organizations can make...
- Charting Your Analytical Future - "Making predictive analytics part of your business processes" Webinar This session will show how predictive analytics can be used throughout the organization by anyone looking for answers and how organizations can make...
- Capturing Data in Motion: Delivering Real-Time Insight from Data Streams This webcast will help organizations of all types and sizes learn about a technology and business strategy for tapping into the wealth of... All Business Intelligence/Analytics White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!