We’re not quite in the trough of disillusionment yet when it comes to big data, but certainly there are numerous stumbling blocks. As with almost all new technologies, after the initial period of infatuation, many have found they are still far from realizing their visions. In fact, quite a few have realized that they never actually had a vision—their big data implementation has not been driven by an end purpose but by the desire to play with cool new toys.
Organizations which have achieved new breakthroughs using big data—whether it is accelerating insights in clinical research by ten-fold or delivering personalized engagement to millions of customers—have a few common qualities, the least of which is technical mastery of the raw horsepower involved. And while having insightful data scientists on staff is a key to success, it does not guarantee it.
To achieve an end vision utilizing data, big, small or otherwise, organizations must agree on its meaning and purpose. And that requires two additional elements—metadata and EQ (emotional intelligence quotient). While the first is a logical yet often neglected topic in big data conversations, the latter is hardly ever considered.
Why metadata and EQ? Because with any important data project, different people and functions across the organization have to first agree on definitions and semantics. And in the big data world, that requires a new, dynamic approach to metadata, as well as a highly refined communication, negotiation and even conflict resolution skills.
Metadata of course encapsulates the structure and meaning of data. But given the level of complexity and variety now tackled by big data, the old approaches to metadata (remember the old enterprise metadata repository?) break down. If humans have to sit down and manually figure out and define all the metadata, it’s a losing proposition. Metadata has to be managed in a highly dynamic manner, where the data platform is constantly sensing the changes in data structures and relationships, and leveraging machine learning to help infer patterns and semantics. And while this is an area where many teams can mature their practices significantly, metadata is always near and dear to the hearts of data geeks.
But no matter how advanced the metadata technologies are, they alone cannot define the meaning of the data. At the outset of a data project, people will inevitably disagree on how to define the data—not due to politics, per se, but based on legitimate business reasons. Should the order value be based on a fixed or floating currency exchange rate? Is a customer still a customer if they’ve been inactive for 17 months? Is an “account” a ship-to site, a logo or a corporate DUNS number? And if the basic definition and meaning of the data cannot be agreed upon, it’s nearly impossible then to achieve alignment on the more important question, namely what is the purpose of the data?
It takes a lot of EQ to navigate these differences and create a common framework that binds rather than divides. Yet how many interview guides for data scientists probe their emotional and social savvy? To have big data projects with big impact, yes, teams should make sure they have advanced data platform, but more importantly, they should ensure they have the people skills necessary to motivate the common purpose.