Managing the Digital Data Explosion To Achieve Success

Making data-driven decisions is increasingly complex, given the digital data explosion we’ve experienced in recent years. Analyzing the mountain of data requires the people, processes, and technology necessary to derive insights from all the noise.

Don’t believe me. Here are some stats:

• The amount of data available for analysis grows 20% each year
• Current estimates suggest that by the end of 2025, 181 Zettabytes of data will exist
• 80% of data is unstructured, which is harder to analyze
• The velocity of data — the amount of data generated per minute — has increased significantly

Managing the digital data explosion

Many companies did a relatively poor job of analyzing data before the digital data explosion. For instance, a local grocery chain used scanner data to send automated reorders to vendors but never linked loyalty club data to the items purchased, which would have allowed them to send personalized offers to interested customers rather than mass-emailing offers of no interest, leading them to ignore emails from the chain. The marketing value of using this data far exceeds the logistical value.

And, they had the data!

With the digital data explosion, brands are literally drowning in data they don’t or can’t use. This requires companies to establish a process, acquire the right people, and purchase sufficient technology to effectively derive insights from the data they can access, so they can lap the competition. This is a comprehensive guide that steps you through the process. 

These are the six steps required to effectively manage the digital data explosion that allows you to interpret and make data-driven decisions that boost your performance.

Establish a strong data foundation

The quality of your decisions is only as good as the quality of your data and a lot of data out there is bad data. Either the data is false, misleading, or a mistake. The trick is to clean the data, if possible, and then decide which data is valuable and which is false. This requires the following:

Good data governance through clear policies about data accuracy, security, and privacy to ensure the data is useful without devulging personally identifiable information without a clear benefit to the organization and with the full adherence to data privacy statements to consumers.
Integrating the data across the organization rather than housing it in silos, such as a CRM, ERP, or email system. This provides a holistic view of the data.
Managing the quality of the data through regularly cleaning and validating data to remove errors, duplicates, and inconsistencies. Setting up data input to require necessary fields and limit the data input into these fields (such as only numeric or only alpha data and limiting the number of characters, if appropriate).

Leverage the right tools and technology

You’ll need buisiness intelligence tools, such as Tableau (for data visualizations), Cognos (for dashboarding), GA4 (from Google Analytics, which brings together all your data from the app store, Google Ads, and your website to allow you to create powerful reports), and Hotjar (creates heatmaps on your website to see where visitors are looking). You’ll also want powerful data analytics tools for predictive analytics that go beyond the descriptive analytics available using the other tools, as well as machine learning and statistical analysis, such as SAS. You need to securely store all that data in a cloud-based warehouse. With the digital data explosion, this can get costly, although Amazon offers an affordable product for small and mid-sized companies.

People are your secret weapon for handing digital data

None of this does you any good when it comes to building the insights needed for data-driven decision-making. You need people with the skills necessary to use them, people able to interpret the data to inform their decisions, and folks willing to ask the right questions to get to the right answers. That means you not only need business analysts, but a data-driven culture, where decisions rest on data not hunches. Since data interpretation is part art and part science, reward experimentation and new ideas, even if they don’t all pan out.

Employ a structured decision-making process

Start by defining the problem or opportunity you face. Then, define the key metrics (KPIs) that measure your performance against the goals you set based on them. Ignore vanity metrics that might look good, but don’t strongly correlate with success. Collect relevant data from whereever it exists into one place using interactive dashboards that make interpretation much easier. Using these insights, make logical decisions, then review the results to improve your decision-making process.

Digital data sources and connections

Obviously, data used for marketing purposes comes from a variety of internal and external sources, including phone calls to customer support, website visits, sensors, email marketing programs, internal sales data, a CRM system, an ERP system, or any one of other available sources. You might also incorporate public data, such as economic data and data from other government sources to improve predictions.

The problem with obtaining data from such a wide range of resources is that the format might vary and you must correlate customer data from different systems using a customer phone number, IP address, form completions, or other customer information. Changing the formatting likely requires programming, since big data can’t be processed manually.

Once all the data is cleaned and collected into an interactive dashboard using data visualizations to improve interpretation, it falls on decision-makers to understand how the data fits into the opportunities presented using statistical tools to build predictions to guide their decisions.

Protecting access to saved data from unwanted invasions, even from internal staff without a legitimate need for that data, is also challenging, as you might want staff to freely access some of the digital data stored in the cloud so they can make decisions relevant to their tasks, without giving them access to everything.  Cybersecurity is paramount, as a data breach can be disastrous for your company and the personal information of those whose data you collected.

Improving insights from digital data

Unfortunately, there’s no magic bullet that works for every firm, every type of data, and for every data use.

Medical devices produce a type of data that has multiple uses, for instance. It might need to go into a patient’s electronic medical record (EMR) or the data might be needed in an aggregate form to aid scientific inquiry or manage the maintenance of the device — or all 3.

Banks and other financial institutions have a host of laws that they must follow, so tracking data should support their adherence to policy and law. But, they also need to manage customer accounts and understand the bank’s financial position.

As in our grocery example above, grocery stores need to manage scanner data for both logistics and marketing purposes.

McKinsey Global discovered:

… retailers exploiting data analytics at scale across their organizations could increase their operating margins by more than 60 percent and that the US healthcare sector could reduce costs by 8 percent through data-analytics efficiency and quality improvements.

Yet, the companies achieving these results, like Amazon and Google, are the exception, rather than the rule, according to McKinsey. Most firms see returns on the order of 1%. Is this a failure of big data or the firm’s use of big data?

The McKinsey study shows it’s mainly a function of not being able to scale analytics, so firms only use a slice of the business to apply big data or only do so on a test basis. It’s like they’re afraid to gamble on big data — and forgoing 59% of the return they could experience with effective analytics.

In their study, here are the reasons firms aren’t getting great ROI from analytics:

1. Managers prefer data mining for diamonds rather than engaging in a sustained, scientific data analysis effort.
2. Front-line managers fail to understand analytics, seeing results as “black box” tools that don’t generate faith in the findings. Hence, insights don’t translate into improved decision-making.
3. Legacy systems within the firm actually impede the use of analytics insights. It’s not an accident that companies like Amazon, which were formed around big data, thrive in the digital data explosion while their competitors simply drown.
4. Data-driven firms have pushed decision-making down in their relatively flat organizations, while their competitors flail about awaiting approval from bloated organizational hierarchies.

Challenges to thriving in the digital data explosion

First, firms need to hire skilled analysts to manage digital data. Only 18% of companies believe they have the skills to effectively analyze data. Unfortunately, there’s a serious shortage of trained BI (business intelligence) employees. And, in many cases, the few folks with the skill set to perform analytics are out of computer science and lack the understanding of business processes necessary to drive decision-making based on analytics, and business schools have fallen short in adopting BI training in their curricula.

And, the salaries offered don’t match the skill set required. Hence, firms end up hiring folks inadequately trained in analytics with far too little practical experience to provide necessary insights.

Solution: hire or train those with business process understanding to manage data analytics

Secondly, organizational structure impedes the use of analytics to drive decision-making.

Solution: Empower front-line employees to make decisions based on real-time data. Management experts argued for decades that flatter organizations were needed to achieve success. Data analytics makes this imperative.

Third, organizations rely on data and analytics that are anemic, especially in marketing. Over-reliance on vanity metrics and descriptive data handicaps managers trying to improve their ROI with better insights.

Solution: Use predictive analytics (and prescriptive analytics) to guide decision-making and allocate resources. Algorithms rule the day. Build advanced analytics into the DNA of the firm.

Fourth, organizations fear that automating activities might be inaccurate.

Solution: face it, machines are better at managing data than humans and make far fewer mistakes. For instance, the US government decided to slow down the development of a self-driving car after one had an accident. Humans have accidents every day and auto accidents are a leading cause of death in young people. Maybe they’re not perfect, but machines outdo humans when it comes to performing routine calculations.

Solutions to managing big data

Firms need to focus on hiring translators — people trained in analytics and business processes, for example. These people don’t come cheap. In 2023, the median salary for these folks was $108,000, which doesn’t include benefits, bonuses, and other expenses associated with hiring these highly trained individuals. There’s also a shortage of these folks, which will drive up salaries and make it hard for smaller firms to find trained BI staff.

Next, they need to overcome the organizational and cultural obstacles to using big data insights.

Make insights easier to understand and act upon, according to an IBM study that found this was a major deterrent to using big data. User interfaces for most tools are awkward and require a huge commitment to understanding how to set up a problem. For instance, SQL has its own arcane language. When SPSS and SAS translated their languages through a series of drop-down menus, analytics became more accessible and understandable. SQL and other tools need to take a lesson.

Start by asking questions to solve business problems rather than seeing what the data can tell you. Instead of building insights, firms are too busy chasing their tails in data management and cleaning.

People are visual and looking at tables of numbers is mind-numbing. Instead, transform data into visual representations that ease analysis. Pie charts, line graphs, and other visualizations democratize analytics and speed insights, according to IBM.

Conclusion

Managing the digital data explosion is complicated, requiring a culture of data-driven decision-making, the right staff, the right technology, and the right processes. However, the rewards from successfully implementing a process for making sound decisions based on data, rather than hunches or doing what has been done in the past, are astonishing.

Need marketing help to support business growth?

We welcome the opportunity to show you how we can make your marketing SIZZLE with our data-driven, results-oriented marketing strategies.  Sign up for our FREE newsletter, get our FREE guide to creating an awesome website, or contact us for more information on hiring us.

Hausman and Associates, the publisher of MKT Maven, is a full-service marketing agency operating at the intersection of marketing and digital media. Check out our affordable coaching programs to help your business thrive into the future.