What’s the big deal with Big Data?

Big Data is the technology focus of the year, and Hadoop is the software that is gathering all the attention of investors and startups. How much is hype and how much is ‘wave of the future’? There is always some amount of hype around new ideas, but I’ll argue Big Data has its value based in diversity.

Diversity of locations

Those who argue that Big Data is just hype (and there are a few) ignore the argument that there is a new diversity of data that has never existed before. The diversity comes from the combination of internal and external (and cloud), mobile, plus the explosion of personal data from social networks. 800M people have put their preferences and relationships on Facebook, but there are many, many other places where data is now available that was never reachable before.

Diversity of users

Big Data has benefited large organizations that could afford hardware and quantitative analysts for many years, but has been out of the reach of small to medium companies. The low cost of storage and open source software has made this technology available to a broad spectrum of companies. It will only get cheaper, meaning the adoption will increase.

Diversity of data sources

The proliferation of web-enabled devices means that satellite, RFID, machine-generated and many other sources of data are in the mix along with consumer information. This data, taken at the right volume and velocity, demonstrates patterns that are useful for predicting opportunities and problems in supply chains, contagious diseases, drug therapies, and a host of other meaningful ways. It opens the door to ideas and solutions that weren’t imaginable a short time ago.

Data traction

Big Data without context or purpose isn’t all that useful, and that’s where process management comes in. In Clay Richardson’s words, Big Data needs Big Process. In his post, Big Data Ain’t Worth Diddly Without Big Process, he argues that ‘Big Process’ is a move from silo’d approaches to BPM and process improvement to more holistic approaches that have a chance to effect business transformation.

Companies have long brought together subject matter experts and mapped how things should be done, but we’ve never really been disciplined about applying the wealth of available data in a systematic way that can revolutionize business. You can’t improve what you can’t measure, and Big Data is all about having the right, timely information to measure meaningful things and affect change.

That means process becomes less about rigid flows and more about an organization being able to use data in specific ways to improve product development, strengthen marketing and sales, and create better customer service. Big Data used well drives organizational change, and that change needs to be wisely managed as an agile process infrastructure.

The companies that don’t put their ear to the tracks by managing Big Data are the ones most likely to be disrupted. And there is lots of disruption ahead.