Welcome to the new kid on the block, same as the old kid but sharper, tougher and even more relevant to business than before. Who can it be, what do we call her and where does she live?
Well here's a surprise. The newcomer is none other than the 4th generation of Enterprise Data Warehousing. Ready and prepared to elevate Big Data, Analytics and an accompanying wealth of must-have data accessories to the dizzying heights of value-adding, business-changing and stakeholder-pleasing usefulness, utility and usability.
That’s right. Just when you thought Big Data had dropped out of Gartner’s hype-cycle and into the recycle bin of disillusion, soon to be despised, ridiculed and then forgotten, along comes the next generation of Data Warehousing to save its bacon.
So, I hear you ask, where am I going with this?
I wanted to tell you about an incredible new opportunity for Data Warehousing that will put the max well and truly at the centre of the maximisation of business value obtained from leveraging business data assets.
We all know that Data Warehousing, when done right, has delivered benefits for those businesses that can adequately align with a centralised yet responsive data logistics solution. We can also note the ample evidence that clearly shows that subject-oriented, integrated, time-variant and non-volatile Data Warehousing provides superior degrees of agility, adequacy and timeliness and to levels that other data logistics and supply solutions can only dream about.
Traditionally, Data Warehousing received, stored and supplied reference and transaction data originally created by business applications and operational systems. We loaded up our Data Warehouses with data such as customer information, product information, employee data, general ledger data, account information, and sales and orders. Other data fed into the Enterprise Data Warehouse has come from external data providers, such as those who sell and deliver price-curves for stocks, commodities and other financial instruments, as well as from providers of data such as foreign-exchange rates, weather forecasts and risk indicators of all types, colours and shades. Many businesses, armed with such data, have been able to compete exceedingly well.
Then along came Big Data. Although it has been around for many years, Big Data has only recently come to public prominence, probably due to a hodgepodge of complimentary and conflicting reasons. Big Data comes at us from all directions in massive network-shaking volumes, in a rich-rainbow of varieties and at alarmingly vertiginous velocities. Some of this data can actually be quite useful for business. Which is the part we are interested in.
Many attempts have been made to work out how to align Big Data and Data Warehousing, and just with all new things we have managed to formulate and discuss a wide range of bizarre, wacky and plain-wrong answers and options before arriving at the more coherent and cohesive ones. Now, after a brief hiatus full of confusion, misunderstanding and exaggeration I think we have the fundamentals pretty much nailed, for now.
Part of it boils down to analytics. I will make a little analogy. Imagine there are two types of analytics: Kleenex analytics and silk-handkerchief analytics. The Kleenex analytics come into play when we are getting web-site page feedback information thick and fast and we customise what the user sees based on what they are hovering their mouse over, clicking on or gesturing at. It’s pretty much about getting the data, doing a quick and dirty analysis, matching with anything that needs to be matched with, formulating a response and then adjusting the user-experience, in near-real-time. The idea is simple. Put stuff in front of the user that they will notice, remember and buy.
Silk-handkerchief analysis goes deeper, is far more extensive, and its outcomes and insights can be measured, used and applied in terms of anything from seconds to years, and encompass individuals, groups and whole markets. In this type of analysis, we can look at the entire life-time of sessions, extended behaviour and usage patterns, or indeed, complete collections of sessions. We can derive data that can then be sliced and diced in almost anyway we would like to and at any level of abstraction, in order to derive insight, passive, active or hybrid. This comes in addition to traditional awareness such as life-time-value, enhanced profitability forecasts and projected customer-loyalty, and it is where the 4th generation Enterprise Data Warehouse approach comes into its synergistic own.
For example, by simply complementing Inmon’s four rock-solid defining characteristics of Data Warehousing with four simple engineering abstractions (in terms of context, view-periods, extended-classifiers and qualitative-quantitative data integration) we can set ourselves the incredibly invigorating, empowering and creative mission. One that allows us, Star-Trek like, to “explore strange new configurations, to seek out new relationships, patterns and insights in data, and to confidently go where no data supported challenge and strategy has gone before.”
Here’s an example. Just imagine that a star trader on your commodities floor has made a trade that no one understands and quite a few think is rather crazy. Well, what if that bet turns out to be a great call? With an Inmon Data Warehouse (with the aforementioned abstractions) it will be possible, for example, to go back in time, and replay what could be seen from multiple perspectives: the periods leading up to the trade, the moment of the trade itself, and after the trade has been made. Looking up, backwards and forwards over a continuous period of time. It will give you answers to questions such as "how did he do that?" and "what was he looking at?"
Not only that, but now with the accelerated evolution of textual disambiguation and Textual ETL, we can augment such data with structured intellectual capital that could, for example, tell us more about why certain decisions were made in the moment when they were made.
For me this is yet more proof that as data logistics and decision support platforms go, it’ll be a long time before the Data Warehouse (now in its 4th generation) can be even remotely challenged, especially where it is most strongest: delivering adequate, appropriate and timely structured business data to support strategic and tactical business needs.
Many thanks for reading.
I shall be expanding on these themes in future pieces on the IT Circus blog here at IT World, so please let me know if there is any aspect that is of particular interest to you.