Big data to drive a surveillance society

Computerworld - NEW YORK -- As real-time and batch analytics evolve using big data processing engines such as Hadoop, corporations will be able to track our activities, habits and locations with greater precision than we ever thought possible.

"It will change our existing notions of privacy. A surveillance society is not only inevitable, it's worse. It's irresistible," said Jeff Jonas, a distinguished engineer with IBM. Jonas spoke to a packed house of several hundred people Wednesday at the Structure Big Data 2011 conference here.

For businesses, the ability to determine where people are by using geo-locational data will help them personalize advertising and marketing materials disseminated via the Web. For example, if a company knows a customer is in Aruba, it won't bother showing him ads for restaurants in New York; it might market sunblock or scuba-diving excursions instead.

Knowing where people are will also enable companies to accurately determine which potential customer is which. For example, if there are five people in the U.S. who have the same name and the same date of birth but live in different cities, it would be possible to verify the identity of each individual by determining their locations at a given time.

"Just look at the last 10 years of address histories ... it is very telling if this is the same person or not," Jonas said. "Two different things cannot occupy the same space at the same time."

Jonas said 600 billion electronic transactions are created in the U.S. every day, and many of those transactions come from geo-locational data generated by cell phones, which through cellular towers, triangulate a person's exact location at any time. Wireless providers have that data in real time.

By looking at data over a period of years, corporations can know how you spend your time, where you work, and who you typically spend time with.

"This is super food [for big data analytics]," Jonas said. "With 87% certainty, I can tell you where you'll be next Thursday at 5:35 p.m."

"Big data" -- an industry term that refers to large data warehouses -- includes machine- and human-generated data such as computer system log files, financial services electronic transactions, Web search streams, e-mail metadata, search engine queries and social networking activity. In 2010 alone, 1.5 zetabytes of that kind of data was created, most of it machine-generated. Companies filled their data center storage systems with about 16 exabytes of that data last year, according to Jason Hoffman, founder and chief scientist at cloud software provider Joyent.

Bill McColl, CEO of analytics engine vendor Cloudscale, said that up until now, big data analytics has been about offline queries or "MapReduce" algorithms, which were developed by Google. But 90% of corporate data warehouse users say they want to move forward into a world with real-time analytics.

"Companies know if they can extract more insight from data faster than their competitors, they're going to win," McColl said.

Jim Baum, founder and CEO of Netezza, maker of a massively parallel processing (MPP) data warehouse appliance, agreed with McColl. Baum argued that if a corporate user has to wait even three days to get an answer to an analytics query, the user won't bother asking a follow-up question that could the key to unlocking the truly valuable insights the information has to offer.

"If I can get an answer in real time, I will ask the next question and the next question, and that'll be followed by another. Getting answers in near real time is critical. It's the enabler of what we can do with big data," said Baum, whose company was purchased by IBM last year. IBM's purchase of Netezza was among a flurry of big data analytics vendor acquisitions over the past year. Other deals included EMC's purchase of Greenplum, Hewlett-Packard's purchase of Vertica and Teradata's planned acquisition of Aster Data Systems.