You’ve probably heard the stats. Things like, “more data is created every day than existed in the entire world back in 2003” (true). Or how about, “72 hours of video are uploaded to YouTube every minute”[1] (true). Impressive as they are, these kinds of stats share another important characteristic: they are totally useless. When I hear them, I get annoyed – like being at a dinner party and being forced to listen to that guy who needs to impress everyone with his knowledge of random trivia. Here’s a stat for you: “if you strung together all the useless ‘big data’ stats quoted in the past year, you could build a road that would send those so-called experts on a one way trip to Mars,” (unverified). Yes, Harvard Business Review called statistics sexy[2], making the mathematicians among us quite happy, but it sure does seem somebody has let it get to their heads.
The problem I have with the way people talk about big data is that the focus is on the data. I hate to break it to you, but I already know what big data looks like. I have dozens of spreadsheets on my computer that have thousands of rows that I can’t comprehend. Telling me about another billion rows of data (or whatever), doesn’t actually help. It’s like the US national debt. I get it. It’s too big to comprehend. Now tell me something I can do something about.
The fact of the matter is that we all interact with big data all the time, in many cases without even knowing it. Take all that real-time traffic information you need to get to work on time. Google is aggregating it from the movement of all the smartphones of all the people driving in all the cars on all those roads ahead of you. But you don’t see the data - you see green, yellow, and red roads. Even then you don’t see them as colors; you see them as meaning – the meaning you need to make a decision, in this case the decision on what route to take. This is just one of dozens of examples I could give.
The best big data solutions don’t look like data. When big data solutions are done right, the numbers and stats disappear and blend into the background. You shouldn’t see data, you should see meaning. What does this look like? There is an emerging paradigm that comes from our consumer experience: Need directions? Looking for a place to eat? Want to edit a video? There are apps for that. They are not monolithic tables of numbers. They are as varied as the questions that need answers. They are visual and interactive. They use data but they don’t look like data.
Let’s acknowledge that building statistical models is just a small first step, the easy part if you will. The hard part then is to turn data into meaning by building apps: to embed them into a business process, to present them in a way that generates greater human understanding, and to use them to help business users make real decisions.
Sorry to pull the rug out from under the mathematicians out there; I know we don’t get called sexy very often.
As published by the Insurance-Canada.ca blog, The Intersection, November 16,2012.
http://www.youtube.com/t/press_statistics
http://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/ar/1