The problem with Big Data: It’s Like Teenage Sex

I’m not sure where this quote about big data is like teenage sex came from, although an internet search assigns it to Dan Ariely at Duke University. Regardless, it really hits the problem of big data on the head — it’s important, confusing,  and it’s hard to do right. Maybe we can carry the analogy a little further to say big data also requires experience, not just reading a book.

Today, I’d like to explore the problem of big data, why you need to develop your capabilities to analyze big data, and how to use it correctly.

What’s the big deal about big data?

It’s not just big, it’s massive.

It’s not just big, it comes in various formats — text, numeric, image; some of which are easier to analyze than others, but all are critical to your success.

It’s not just big; it’s spread over different tools and properties, which makes it hard to associate related data across data sets.

It’s not just big; it’s hard to determine cause/ effect. Correlation doesn’t equal causation, so confusing the two leads to poor decision-making.

Here’s a quote from Big Brother, IBM, about Big Data:

Big data is arriving from multiple sources at an alarming velocity, volume and variety. To extract meaningful value from big data, you need optimal processing power, analytics capabilities and skills.

Let’s delve into this a bit.

The problem with big data

Actually, the problem with big data is that it actually creates multiple problems for organizations that must remain nimble and use their data to make better decisions that allow them to improve performance.

Velocity

Big data comes at you very fast, and that speed increases as more devices get connected to the Internet (for instance, the Internet of Things -IoT). Marketers must become more adept at streaming data in real time from sources like Facebook, along with developing algorithms and dashboards that allow them to analyze the data quickly. Data visualizations and interactive dashboards significantly aid these efforts.

Volume

We’re drowning in data — data from our sales channel, from social media channels, from our sales force, from our CRM, from our newsletter, from market research, from industry studies … And IoT brings massive amounts of data from our connected homes, connected devices like smartwatches and activity trackers, and medical devices. Just keeping up with the volume of data streaming in requires significant planning on how to organize it into databases, how to store it, what to store, and how to make associations across databases, let alone how to make sense of all the data.

Collecting all this data is useless unless you have a plan for analyzing it. And, while the cost of storing data dropped significantly with services like Amazon’s S3, storing terabytes of data is still expensive.

Variety

Variety might be the most challenging part of big data — especially when you consider the vast amount of unstructured data in the form of video and images that comprises much of the data generated by consumers. IBM estimates 80% of data is unstructured and, unlike structured data, the tools available to analyze it are limited.

Veracity

I’d also like to add my own V to those mentioned by IBM — veracity. Too often, the data we collect is suspect for one reason or another. Among the reasons we might doubt the accuracy of our data are:

• Did we accurately assign sentiment to the utterance
• Was the data itself accurate
• Was the structure of the data maintained during collection, or might the data be corrupted or misaligned to cloud our variables
• Were the data truthful — ie, did consumers reflect accurately
• Was the data representative, or do we have skewed data

When we talk about veracity, data analysis becomes truly scary. If we try to make sense of data that’s corrupt, or worse, make decisions based on this data, we’re as likely to be led astray by our data. This clouds our understanding of what’s going on and leads to poor decisions.

Other problems with big data

Now that we have some understanding of big data as well as the problem with big data, we can talk about using big data to improve organizational decision-making.

While I love this cartoon from Tom Bishburne, it’s disastrous thinking within an organization, yet all too typical of the uninformed thinking that is often seen in boardrooms. And, this cartoon underscores the contention that big data is like teenage sex — that people think it can solve all the problems in an organization. Big data won’t solve your problems; it’s a tool that might lead you to a solution if you have the right knowledge and tools to analyze the data.

I’m not alone in thinking there’s a problem with big data. Here’s what other leading experts have to say about big data:

Here are just a few of the top blogs bashing data with respect to what it’s done for marketing:

• Squeezes all the creativity out of marketing [Convince and Convert]
• Leads to inaccurate marketing strategies [Marketing Tech]
• Doesn’t offer trustworthy insights for marketers [MarketingLand]
• Data and predictive analytics are getting in the way of doing basic marketing [Forbes]
• Lied about election results, so why should we trust it to help with marketing [Huffington Post]
• Doesn’t belong in marketing’s future [Marketing Tech News]

Some of these problems are the result of poor analytical skills or improper use of big data. Still, likely the biggest problem with big data is that firms lack the resources and staff skills necessary to derive meaningful insights from the data.

Another problem is misusing big data, which is a much easier problem to fix. Here are some great uses for big data:

Uses for big data

Deliver customer insights

A sure means to business success is giving customers what they want, when they want it, at a price they deem right (not cheap, but a price they see as providing value). Big data offers customer insights that can help us deliver better customer service and achieve higher levels of satisfaction. For instance, going beyond sentiment analysis, social platforms offer a plethora of customer insights, including unmet needs (that suggest new products) and customer service problems.

For these insights, you need to look beyond the mean or everyday conversations to look at the outliers who offer a unique perspective. I’ve worked with firms that tend to dismiss comments when they don’t occur frequently, assuming they are isolated. That may be true, or it may NOT be true. If an isolated comment is just the tip of the iceberg, it is the forerunner of major trouble lying ahead for the firm.

Rather than dismiss stray comments, investigate them. Even an isolated complaint can spiral out of control, while a quick response usually satisfies the complainant and reduces any negative impact it might have caused. Every complaint is an opportunity to turn it around, if you’re listening carefully.

Deliver targeted communication

Nothing fails so fast as communications that aren’t appropriate. For instance, I liked getting suggestions from Amazon when I was searching for a new pair of hiking boots, but was annoyed when they continued sending suggestions even after I purchased a pair on their website. It told me I really didn’t matter to Amazon because they weren’t listening to me.

The same thing happens on LinkedIn all the time. I get messages from folks offering to help market my businesses. While some might agree I need help, as a marketing professional, there’s no way I’m hiring someone to market for me, since I have the skills and experience needed, plus I’m more familiar with every aspect of my business. All you have to do is take a cursory glance at my profile to see I’m another marketing professional, so offer a customized solution beyond my ability if you want me to consider your business.

Delivering targeted communications is a little tricky, especially with big data, where the data itself comes from a variety of sources. Combining insights for individuals requires using a key to merge different databases together. That’s where using social logins really helps — it acts as the needed key to merge data into a single record and offers improved insights.

Optimize performance

Big data generates insights to help optimize performance. For instance:

• An ambulance company used data about ambulance requests to distribute resources and schedule staff to reduce ambulance response time, resulting in fewer deaths or complications and better satisfaction without increasing expenses.
• Power companies use data insights to manage demand, which reduces the need for new, expensive power generation plants.
• Amusement parks use data to determine staffing and adjust pricing to reduce demand during peak load times.
• Airlines use data in the same way to determine optimal pricing based on current and historical booking levels between cities. This is the basis for demand pricing.

Lots of companies use data to optimize performance and manage demand so as to improve satisfaction without increasing costs, or with minimal impact on costs.

Using big data

Building insights starts with collecting big data. Evaluate potential sources of data as input for decision-making. IBM offers a nice infographic of where data comes from.

Collect data

Just because you have data doesn’t mean you should save the data or use it for decision-making. By the same token, data you don’t collect can’t be analyzed. It’s a cost/benefit thing — you collect data only when the potential benefit outweighs the cost.

For instance, asking for a lot of personal information when visitors sign up for your newsletter might be nice to have so you can serve them better, but it seriously reduces the number of folks who will sign up. When the value of having that information outweighs the reduced subscription rate, then go for it. If you already have the information, then asking for it again is a waste. Instead, consider adding a request for additional information in your welcome email.

Clean data

I think this is a step missed by many big data analysts — making sure you have clean data. Garbage is = garbage out.

I usually spot check my data to look for things that don’t look right. If your data is quantitative, run descriptive statistics to look for outliers, determine if the calculated mean makes sense, and ensure you don’t have a lot of respondents with missing data. Spot checking your data is less feasible with unstructured data, where machine categories might not be accurate and some training is necessary. AI using large learning models is easing this somewhat, but it still has a long way to go.

Next, I look at the data as a whole to see if I have some data problems.

Merge datasets

Next, I’ll merge datasets across databases using some type of key, which may be customer number, email address, phone number, or some other type of unique identifier.

Analyze data

In this step, you finally see the value of big data. Be careful, however, or you’ll end up with nonsense. With enough data, you’re bound to find correlations and some of them (potentially most of them) are just junk. Some that might seem like junk are really causal relationships.

Remember the Super Bowl ad showing all the kids born 9 months after the city’s team won the Super Bowl. Correlation or causation? Likely, there is a causation. Meanwhile, the correlation between the economy and the length of women’s skirts is correlated (weirdly), but they’re not causally related. If that were the case, governments would pass a law that women’s skirts could only measure a certain length.

I think it’s important to have marketers trained in analyzing big data because I truly believe you need to understand the concepts behind marketing to identify important relationships among all the junk relationships that appear in big data analysis.

Conclusion

I hope this post gave you an appreciation for big data and the problems associated with analyzing it. I can’t stress enough, however, that the fundamental problem with big data is developing talented data analysts with the skill, experience, and talent for analyzing massive datasets. Remember, data analysis is part art and part science.

Need marketing help to support business growth?

We welcome the opportunity to show you how we can make your marketing SIZZLE with our data-driven, results-oriented marketing strategies.  Sign up for our FREE newsletter, get our FREE guide to creating an awesome website, or contact us for more information on hiring us.

Hausman and Associates, the publisher of MKT Maven, is a full-service marketing agency operating at the intersection of marketing and digital media. Check out our full range of services.