As data gets bigger, time value gets smaller

By now most people understand that data is growing exponentially in volume, velocity and variety. We’re awash in data and the first instinct is to store as much as possible for later evaluation. We store it simply because we can’t manage it in the moment it arrives.

But in what has to be the biggest irony of Big Data, the time value of information, the measure of its useful lifespan, is growing shorter. So just at the moment we struggle to keep up, we find that unless we focus on the data that only matters for a short time span, we forfeit the game.

What’s driving this? Things like mobility, machine data, sophisticated fraud, geolocation, customer retention and more. The decreasing time value of data significantly increases the need to:

• Know that our customer is in our store or on our website while they’re still there to interact and give differentiating service
• Sense that the train brakes are overheating ahead of a dangerous situation and costly repair
• Shut down a credit card before the transaction goes through
• Recognize a customer service issue and react before losing their business
• Spot a dangerous healthcare situation before the patient gets sicker or worse

Decreasing time value has a ripple effect across the organization, as analytics and processes need to speed up to make use of data in its useful lifespan.

This is also the best argument around for why Big Data often needs to be real-time data. This is an in-memory problem not well suited to a traditional database and companies wishing to compete need to recognize and retool to stay competitive.

It will be interesting to see who figures that out and who stays on the sidelines.