Big Data: Insight, Not Integration

In the movie Hoosiers, there is a classic scene where the coach of Hickory, Norman Dale (played by Gene Hackman) wants to calm the nerves of his team before the state championship game. His concern stems from the fact that the game is played in a large stadium, with a huge crowd, media, and other distractions. In order to focus his team, Coach Dale walks the team into the empty gym, pulls out a tape measure, and asks the team to measure the floor, then the baskets. The team promptly reports back that the baskets are 10 feet, just like every other basketball court. The team quickly grasps the message: the baskets are 10 feet and the gym is exactly the same as any court they have ever played on. Therefore, they need not worry about external distractions (gym, court, media, etc), they merely need to focus on themselves (the players) and what they do (they plays).

Big Data has reached the point of maturity that is marked by unusual announcements, with unexpected companies/organizations/parties trumpeting their use of data. Here are some recent ones I've seen:

The interest in exploiting data is not a fad, its simply a reaction to the fact that more data is now available. However, there is one pattern that I see in these examples and many of the organizations that I talk to: The world is much more interested in insight from data, than in tools to analyze data. Sure, tools may help you glean insight, but wouldn't you just prefer to have the insight, without having to look for it? Paradoxically, I see most of the IT industry focused on building tools, data management technologies (Hadoop, etc), and other infrastructure.

The winners in this race will be those that can provide insight, without the need for expansive (and expensive) tools and integration. Like Coach Dale showed his team, its not about where you play, its about what you do when you are on the court. When it comes to Big Data, the world wants insight, not integration.