What we all need to know about Big Data

Watching the buzz around Big Data, it would be easy to dismiss the hype as an answer looking for a solution. After all, how many times have you read about a remarkably good business model (other than Facebook, Google or Amazon) making millions through Big Data technologies?

However, by the time you read about solutions, technology is likely already in mainstream use and past the hype, meaning: If you wait for your competitors to show results, you’re already many months behind and unlikely to catch up soon. You’re eating everyone’s dust.

Growth spurt

Big Data goes well beyond putting new labels on existing products, though there will certainly be a fair amount of that. Big Data is about a maturing set of affordable technologies that take advantage of several shifts that took place in the past couple of years. You’ve heard it before, but mobile, social, and less-known but very important machine data (logs, etc.) are increasing the data ‘haul’ at a rapid rate. Think of this, too…this is happening to organizations that haven’t maximized the data they already have.

New tech

Just in time for data’s growth spurt, we can now cheaply store lots of information through the web, or Cloud. We can keep it in-memory much more easily, too. Add to that an open source platform, Hadoop and its ecosystem, that can distribute storage and processing across many machines. Each of these is a major change in technology, but taken together, are the key ingredients for a new level of finding and understanding patterns.

Good, Bad and Ugly

Patterns give us insight into what’s happening in our marketplace and world that we never would have had through traditional means. Those patterns can be a way to sell better or to avoid disaster.

• Good: From better consumer products to getting vaccines to critical places in the world
• Bad: Learning why things don’t sell or where customer service is failing by ‘reading’ comments and click patterns
• Ugly: Spotting fraud in credit card transactions, movements of money and other illegal activities

When you look at the growth of data, the advances in storage and computing, and the newly interconnected world, there is an unmistakable opportunity to be involved in game-changing ways of doing business. That isn’t hype, even though hype may blur the benefits. Big Data is our new reality.