Being able to harness big data has enabled some incredible advancements in business. Just consider some of the following use cases.
We can now target and schedule communications to hit when they’re going to be most effective. With attribution modeling, we can pinpoint what part of a campaign is performing best (or worst), and optimize mid-flight. Analytics apps can tell us when people visit our site, how many times they visit, their behaviour, and where else they go. We can map social networks and pinpoint influencers.
Big data might even (eventually) be the key to upending John Wanamaker’s age old truth about advertising: “half the money I spend on advertising is wasted; the trouble is I don't know which half”
Here’s the kicker, though – big data isn’t half of what it could be.
The dream of big data
Big data is the ultimate dream of marketers and business analysts: lots of customer data, from lots of different sources, arriving instantly and being quickly translated into insights.
And when we’re talking about data, it could be anything from a customer email from a newsletter subscription to tracking purchases through a loyalty program. It includes any scrap of information that a business can collect about its customers in each channel and at every touchpoint.
Then, businesses turn those insights into money.
First, the obvious ones: marketing and advertising. If you know what your customer wants based on what they’ve purchased in the past, it’s much easier to sell that to them.
Compare for a minute a generic email and one using personalization techniques.
The generic one might say:
Look at our amazing sale!
This example would probably not be very effective, because consumers are bombarded with these sorts of generic messages all the time.
That same email with personalization might read:
Look at our amazing sale on seeds and nuts!
If seeds and nuts were a recently purchased product, this would be more effective. It could be Spanish wine, or baking essentials, or berries. It doesn’t matter.
What's important is the consumer feels like they’re being talked to directly, with a proposition that solves their unique problem. Personalization makes for much more effective communications.
Of course, big data has benefits outside of marketing and advertising. Businesses can use data to identify issues or market trends, and respond to them quickly. For instance, a company that sees decreased usage among its customer base in a specific region could inform R&D, launch new products, time their price fluctuations, regionalize their offering, and generally maximize their efficiency while lowering their churn rate.
So that’s the dream of big data. But often, it’s not the reality.
The reality of big data
The reality is that we haven’t really achieved the potential of big data. Sure, companies have become smarter about using the data they have to make data-driven decisions rather than subjective ones.
And yes, companies have incorporated dashboards and nice software to allow them to see what their data is doing, so they can make decisions based on it.
But the dream of totally integrated data sources, with instant access and quick analysis? Not quite as widespread as it should be.
And here’s where we get to the bottleneck. Our collective ability to gather data has increased a hundredfold over the past 10 years or so. Consumers’ information is so readily collected that we don’t even think about it anymore.
Loyalty programs, search engines, websites, marketing opt-ins, online stores, in-store sensors, apps, and the huge variety of smart tech we have, all means that there are oodles of data available for businesses to collect and buy.
However, for the most part, businesses are still not able to fully realize that data potential due to three key problems:
- Lack of expertise
- Lack of technical ability
- Lack of legal clarity
Lack of expertise
SAS highlights this problem in a one-pager about big data. Specifically, they say that to get the most out of your data visualizations, you need to make sure the person looking at it has the expertise to understand what they’re looking at.
And there are just not that many of those people around.
What’s more, even if you do have a fully trained staff of top notch data analysts, they often won’t be asking the right questions of the data to derive quality insights.
If you visualize processing big data as a funnel, the processors are eventually going to need subject experts to help them translate raw insights (e.g. X correlates to Y) into actionable recommendations.
Plus, that’s assuming that you get to the more semantic stage of translating big data insights into recommendations!
The steps involved in extracting big data from a swirling pool of data feeds is extremely intricate and involved.
All in all, there aren't enough skilled resources available for big companies to fully flex their big data muscles.
Lack of technical ability
This goes back to the complexities of withdrawing data from a collective pool. But first, we need a clear idea of the journey big data takes.
Data is generated at some touchpoint – for example, a smartphone. Then, that data is fed into the cloud. Then, that data needs to be extracted from cloud servers, then translated into something workable, then analyzed, then passed to subject experts in a more refined state again. And then we get insights.
Resourcing problems aside, this is still very hard to do. Getting clean data out of servers, while it can be done with Hadoop, is still difficult. Big data is, by its nature, unstructured and pooled, which make it harder to pull it together.
What’s more, with so much data gathered, it’s increasingly impossible to use even a fraction of what you have, so really you’re not using big data at all, but rather a much smaller data set.
It’s like eating at an enormous buffet – you can’t possibly try everything, so you’re forced to pick and choose.
Lack of legal clarity
Companies today have access to far, far more information than they can use. Put simply, privacy laws across the globe are hopelessly out of date for the world of big data – there are fundamental legal and philosophical questions here that still need to be answered:
- What counts as private data?
- Who can use it?
- How can it be used?
Our current legal structure is just not equipped to answer these questions, and as a result, big data falls far short of its objectives.
Conclusion
The dream of big data will live on; a world where Google can predict the flu, where companies know what consumers want as the consumer wants it, where each and every decision, at every level of the organization, is driven by data – not by instinct.
However, that day is still be a little down the road. We have some significant challenges to overcome first, such as:
- Challenges around resourcing and the ability to ask the right questions
- Technical challenges of pulling out sensible data from an overwhelming collective pool
- Legal challenges around what we can actually do with big data
We expect that, eventually, these problems will be solved. The dream for supersized data lives on, but in the meantime, we might have to settle on something smaller.
