Big Data Demands Big Picture Thinking

Big data continues to amaze with its longevity at the top of the hype cycle. While we still hear the occasional paean to its enormous size and growth, most of today’s stories extol the ability of big data to allow understand customer behaviour, wants and needs, driving some new golden age of sales and marketing, not to mention customer service. You might be forgiven for thinking that the main value of big data is in figuring out how to sell more stuff to people who may not even be aware that they need it. In fact, this is a very narrow view of big data. Indeed, I would contend that it is a dangerous view for business and society as a whole.

For such a widely used term, big data is very poorly defined. Within this article, I’ll use it in the broadest sense, including all types of human-sourced information—from tweets and social media to YouTube videos—and all classes of machine-generated data—from simple sensor data on the Internet of Things to click and event logs generated by sophisticated computing devices. In short, big data includes all “non-traditional” data beyond that created as part of the classical operational and informational processes providing the legal basis of a business. With this definition in mind, we see that big data use goes far beyond marketing; it forms the basis of every disruptive business change currently occurring. Let’s look at a few examples.

Uber, the $50B darling of commuters and investors, is disrupting traditional taxi services based on the use of pervasive and highly accurate location data available from smartphones. Such data, combined with instant rating information is the operational and analytical foundation that enables everything from driver engagement and customer satisfaction to instantly flexible pricing models.

Monsanto, more usually associated with sterile, GM seed production, is also on the big data trail, having acquired The Climate Corporation for $930M in 2013. This added weather measurements from millions of locations and billions of soil observations to their own data collections. They claim 5% improvement in crop yields over two years and significant cost savings for farmers; not to mention enhanced lock-in of farmers to Monsanto products. CTO, Robb Fraley, is quoted: “I could easily see us in the next 5-10 years being an information technology company.”

In the financial services industry, an industry that has long lived (or died) by the amount and value of its data, big data can change everything. After all, what is money today other than ones and zeros? Earlier this year, TransferWise raised a further $58M in funding, raising the prospect of a $1B valuation. Its business model is wholly data-based. By sharing information across borders about foreign exchange buying / selling needs in real time, it avoids actual money conversion in both markets and does so for a fraction of the fee normally demanded by traditional forex companies. The potential for disruption is significant.

Similar stories abound across every industry and region, changing the way commercial and government operate, affecting how customers and citizens interact with them. Such fundamental changes raise issues beyond the traditional scope of IT thinking. They are transforming the pervasive and often unspoken social contracts that have long existed between business and its customers, between the democratic government and the citizens of countries, and between all combinations of businesses and countries. Some of these issues are beginning to attract attention; other still await wider notice. In Part 2, I’ll look at privacy, the first of two important concerns, and tackle the issue of technological unemployment.

This is Barry Devlin's first blog of a blog trilogy on big data. Click here to read the second part.