The big promise of Big Data: What you need to know today

InfoWorld - In the never-ending quest for a competitive advantage, organizations are turning to large repositories of corporate and external data to uncover trends, statistics, and other actionable information to help decide on their next move. Those data sets, along with their associated tools, platforms, and analytics, are often referred to as "Big Data," a term that is gaining popularity among technologists and executives alike.

Although decision-makers have realized there's value in Big Data, getting to that value has remained elusive in most businesses. That's where IT can help, creating services that empower researchers to delve through large data stores to perform analytics and discover important trends. In other words, IT will prove to be the catalyst that delivers on the promise of Big Data.

[ Get smarter about how you handle the explosion of enterprise data with InfoWorld's Enterprise Data Explosion newsletter. ]

Big Data has already proved its importance and value in several areas. Organizations such as the U.S. National Oceanic and Atmospheric Administration (NOAA), U.S. National Aeronautics and Space Administration (NASA), several pharmaceutical companies, and numerous energy companies have amassed huge amounts of data and now leverage Big Data technologies on a daily basis to extract value from them.

NOAA uses Big Data approaches to aid in climate, ecosystem, weather, and commercial research, while NASA uses Big Data for aeronautical and other research. Pharmaceutical companies and energy companies have leveraged Big Data for more tangible results, such as drug testing and geophysical analysis. The New York Times has used Big Data tools for text analysis and Web mining, while Disney uses them to correlate and understand customer behavior across its stores, theme parks, and Web properties.

Big Data plays another role in today's businesses: Large organizations increasingly face the need to maintain massive amounts of structured and unstructured data -- from transaction information in data warehouses to employee tweets, from supplier records to regulatory filings -- to comply with government regulations. That need has been driven even more by recent court cases that have encouraged companies to keep large quantities of documents, email messages, and other electronic communications such as instant messaging and IP telephony that may be required for e-discovery if they face litigation.

Perhaps the biggest challenge facing those pursuing Big Data is getting a platform that can store and access all the current and future information and make it available online for analysis cost-effectively. That means a highly scalable platform. Such platforms consist of storage technologies, query languages, analytics tools, content analysis tools, and transport infrastructures -- there are many moving parts for IT to deploy and look after.