What’s hiding in your data?

I was invited to speak at Forrester’s Digital Disruption Conference in Orlando this week and finished up my time on stage this morning. I was brought in by Forrester Senior Analyst Mike Gualtieri to speak alongside Terradata’s Chris Twogood about the meaning and impact of Big Data.

Before taking the stage, the three of us agreed that the term ‘big data’ isn’t adequate to cover the breadth and depth of what’s happening in technology, but it is the term that at least starts conversations on the same thread.

Our time on staged started off with Mike’s description of Big Data by the usual explanation of the three V’s of velocity, volume and variety. Chris Twogood and I added Value and Volatility based on our experience with our respective customers. That drew a laugh from the crowd as Mike brought up that we were only limited by the number of words beginning with ‘v’.

Hidden gems

One of the topics we covered at length was the question, “What’s hiding in your data?” Examples were cited of data that isn’t visible to the naked eye but becomes apparent when run against algorithms that bring unseen patterns to life. We agreed that the larger and faster the data, the more potentially valuable correlations can be discovered.

Once discovered, patterns can be put to use in better operational decisions, customer service, risk avoidance and a host of other benefits. The key is to have a system and processes that are able to detect and respond.

We agreed that an interesting side effect of this paradigm is the discovery of correlations that aren’t fully explained. We can know the ‘what’ but we may not know (or need to know) the ‘why’ of a pattern…just that it exists and is predictive in nature.

Crystal ball

Mike brought the conversation around to the value to predict, probably Big Data’s most important benefit. The larger the data set, the greater its predictive accuracy. This is the essence of TIBCO’s 2-Second Advantage, the concept that, “…a little bit of information at the right time is more valuable than all the data in the world after the fact. For those in the audience that felt Big Data = Hadoop, we spent considerable time talking about the need for Big Data to influence decision making in tangible ways. Any less is a ‘research project’ and not an operational value.

For our wrap up, we each spent a couple of minutes giving our advice to the developer community needing to be responsive to a business looking for IT solutions. I shared that choosing a platform is key an that the host of open source applications may have an attractive price point up front, but the lack of features and intuitive interfaces slows down implementation and keeps the problem owner from ever being able to use the system. We all agreed that ease of use and business-facing tools were key.

I closed with the importance of choosing a platform that is tightly integrated but loosely coupled  so that solutions can be flexible and ‘move’ on the fly as requirements change and new technology arrives. I declared the monolithic platform to be a solution of the past for cutting-edge problems. The new generation of Big Data solutions is a platform that layers on existing databases and applications.