Computerworld - As tools for real-time and batch analysis of so-called big data emerge, IT operations are gaining the ability to track the activities, habits and movements of customers with great precision.
Experts say many businesses want to better analyze data stored in emerging massively parallel databases like the open-source Apache Hadoop framework to learn where their customers are employed, what they do in off-hours and who they spend time with.
The information could help companies tailor Web-based advertising and marketing materials to specific customers.
"[The trend] will change our existing notions of privacy. A surveillance society is not only inevitable, it's irresistible," Jeff Jonas, a distinguished engineer at IBM, said at the Structure Big Data 2011 conference in New York late last month.
The term "big data" refers to the massive amounts of information collected from machine- and human-generated computer system log files, electronic financial transactions, Web search streams, email metadata, search engine queries and social networking activity.
In 2010 alone, 1.5 zettabytes (1.5 billion TB) of such data, mostly machine-generated, were created. Companies collectively filled their data center storage systems with about 16 exabytes (16 million TB) of that data last year, said Jason Hoffman, founder and chief scientist at cloud software provider Joyent.
Some companies are already finding ways to dissect the rapidly growing data stores in creative ways.
For instance, bookseller Barnes & Noble has started analyzing an explosion of user- and machine-generated data about its online buyers to determine various customer behaviors, such as what percentage of shoppers make book-buying decisions based on their fondness for a particular author, said Marc Parrish, vice president of retention and loyalty marketing.
"Our Web logs on how customers are using e-readers and e-books have produced 35TB of data and will load us up with another 20TB this year," said Parrish.
"We have to decide with analytics on hand how we capture the customer's imagination and how we move forward," he added.
National Public Radio uses a reporting tool from software provider Splunk to track growing data sets to determine website audience usage trends in near real time, said Sondra Russell, a metrics analyst at NPR.
Splunk crawls logs, metrics and other application, server and network data and indexes it in a searchable repository.
NPR had been using the Web analytics engine Omniture, but Russell said she felt like she was jamming log-based data into a client-side tracking system that couldn't handle the organization's increasing volumes of data.
"I just want to know how many times someone listened to a program during a certain period of time," she said. "With Splunk, I had no delays between data appearing in a query folder and data appearing in reports. I can get any number of graphs without weeks of prep time."
Meanwhile, Hadoop, which unlike traditional database management technology is designed to work with multiple data types and data sources, is increasingly used by large banks, advertising companies, life sciences firms and pharmaceutical companies hoping to better leverage massive amounts of data, said Stephen O'Grady, an analyst at RedMonk.
Computerworld's Jaikumar Vijayan contributed to this story.
Read more about Data Center in Computerworld's Data Center Topic Center.
You are previewing premium content. 
Free Insider Guide
