Social Media Analytics: Sentiment Analysis

sentiment analysis

I talked a little about sentiment analysis in earlier blog posts but never got deeply into the topic. Let’s fix that today.

Sentiment analysis mainly tracks the prevailing opinion of your brand and it’s most effective for larger brands, especially consumer brands. For smaller brands, there just aren’t enough mentions of your brand to make sentiment analysis very insightful. For small brands, scouring specific websites, such as Yelp, EOpinions, and other places likely provides more insights regarding prevailing opinions about your brand.

If you’re an avid reader of this blog, you know I take a dim view of sentiment analysis, primarily because it’s a fairly shallow measure of your brand’s success. By shallow, I mean sentiment analysis only gives information about positive (or negative) mentions of your brand, as you can see below.

sentiment analysis

Why should you care about sentiment analysis?

Sentiment analysis is inextricably bound up with reputation management. When customers or prospective customers encounter negative comments about your brand from rating sites, online reviews, or word of mouth, it causes them to think twice before making a purchase and may dissuade them from buying your brand. For instance, consumers may see you only have 3 of 5 stars and determine to buy from a competitor with a higher rating.

But, what happens when the evaluations aren’t as clear? For instance, social media posts might include an evaluation consisting of words rather than a score. Those words carry a lot of weight with potential buyers but it’s harder to categorize. Sentiment analysis requires a systematic process of categorizing these statements so you can act to improve your market performance.

Using sentiment analysis

Sentiment analysis at its best combines natural language processing with artificial intelligence capability and text analytics to evaluate statements found across various social platforms to determine whether they are positive, neutral, or negative with respect to a particular brand. A firm might track sentiment analysis over time to:

  • determine whether brand actions improve or damage sentiment
  • track brand reputation
  • test how marketing efforts affect attitudes toward the brand
  • attitudes toward new products

However, almost all languages feature a lot of ambiguity that makes it hard to interpret the meaning of words. For instance, maybe a user posts:

XYZ brand is badass!

Sentiment analysis must accurately categorize this as positive sentiment, despite the use of the word bad in the post. In other words, the tool needs to understand colloquial word usage (natural language) to know that bad means good in this context. Artificial intelligence comes in when the tool is “trained”. So, let’s say the tool initially categorized the post as negative because of the word “bad” in the post. A human, as part of the training process, goes in and tells the system they were wrong – this is a positive statement. The tool “learns” that “badass” means good and doesn’t make that mistake again – at least not as often.

How to measure

Now, I could spend several posts showing you how to calculate sentiment using advanced statistics and tuples (don’t ask – they give me a headache), but I assume you just want to use them, not create a tool to analyze sentiment. Hence, I’ll talk about tools already available as a SAAS (software as a service).

There are lots of tools out there for measuring and tracking sentiment. SAS and IBM make excellent tools, but they’re expensive and a little unwieldy for the novice – designed more for business intelligence specialists. Salesforce.com (marketing cloud), Traackr, and ViralHeat all have sentiment analysis or include sentiment analysis as a piece of their social media automation. These tools are much more intuitive and user-friendly than SAS and IBM, which offer more power and flexibility at a much higher cost.

Problems

The biggest problem with sentiment analysis is that you only get an assessment of positive, negative, and neutral. That isn’t good enough from a marketing perspective. Although you can assess sentiment over time to map changes, you really don’t have all the tools needed to make major improvements in sentiment. For instance, should I change feature A or feature B to get the most improvement in sentiment?  Or, should I offer option A or option B? I can’t answer these questions because I really don’t know WHY my target market has a positive or negative assessment of my brand.

A second problem emerges and that’s that, no matter how good, an effective system must still be: 1) trained and 2) listen.

Training

No tool is able to accurately score sentiment right out of the box and some are easier to train than others. That’s because sentiment is unstructured data so the system has to be trained to determine if a particular statement is positive or negative. There’s also the issue of slang, misspellings, and weird sentence structure, which further complicates sentiment analysis. Training can be very time-consuming and tools vary in their ability to be trained, some can’t be trained at all. Across major tools, the industry average is about 65% accuracy – not so good in my opinion.

Hearing

Hearing is the other problem with sentiment analysis. Some tools only listen to conversations on certain social media platforms, although the ones I listed above listen across Facebook, Twitter (now X), and maybe a few other social platforms. Listening to blogs is a little more complicated because the posts are longer. Google Alerts can handle that, but I’ve not found them proficient at detecting all the conversations going on out there. Still, setting up an alert for your brands, key personnel, and some competitors makes sense. Google Alerts must be hand-scored or use a text analysis program, such as Nvivo or HyperRESEARCH.

A colleague interviewed for this post told the unfortunate story of hiring an intern to cull through the alerts and categorize them on a spreadsheet so the client could respond to unfavorable mentions. Unfortunately, this proved too big a task for the intern or Google. Thus, the client, a pharmaceutical company, was unaware of problems with one of their drugs until contacted by the FDA. Not exactly what the company wanted from their listening program.

Most work fairly similarly, so let’s talk about Traackr since I have the most experience with that platform.

Traackr lets you select keywords related to your brand. So, if I were using this tool to monitor my business, I’d use keywords including my name and company name, if I had brand names, I’d use them, as well as names of key competitors, and industry terms such as social media analytics. Traackr would track these keywords, calculate sentiment related to me and my brand, and chart changes in sentiment over time. Traackr has some nice features, in that it also gives you information about users mentioning your brand, thus combining some elements of influence analysis with sentiment analysis.

Other problems

Why negative sentiment?

While interesting, and potentially creating some value, this type of analysis doesn’t tell you much about WHY folks feel the way they do about your brand. Results also don’t say much about HOW positively or negatively people feel about your brand — so real anger gets the same valence as mildly annoyed. Sure, you get an analysis of sentiment change over time but how valuable is that when crazy angry gets coded the same as mildly angry? All the software does is count the number of positive statements and negative statements.

Uncovering problems

Sentiment analysis is also a result measure. Not only doesn’t sentiment analysis tell you why people feel good or bad about your brand (although you can do a deep dive into actual conversations to discover this manually), but you’re finding out about the problem AFTER the emotion occurred when it’s much harder to fix. It would be nice to know there was some problem so you could fix it BEFORE customers started sharing their negative emotions about your brand.

Setting up a system for collecting feedback from customers can fix this problem. If a customer is dissatisfied, you now have a chance to fix the problem before they share their negative experience.

Collecting data

Existing sentiment analysis tools don’t collect EVERY mention of your brand. Most do a fairly good job on social media sites like Facebook and X, but gathering information from blogs and news sites is much more limited. So, you don’t have complete information going into the analysis. That said, I don’t know of ANYTHING capable of collecting every online utterance.

As a manager, you must recognize that the absolute number of positive or negative utterances isn’t a true reflection of how folks feel about your brand. So, choosing to ignore a “few” negative comments is dangerous. Those few comments might represent hundreds or thousands of similar comments your sentiment analysis tool didn’t find.

Accuracy of sentiment analysis

While I’ve always questioned the accuracy of sentiment analysis results, a comment left on one of my earlier blog posts points to even less accuracy than I’d ever assumed.

Recent experiments suggest sentiment analysis data is LESS accurate than a coin toss (accuracy 50%). That’s really scary if your brand makes strategic decisions based on sentiment analysis.

If you’re a skeptic like I am, here’s the proof:

Walid Saba is a professional in AI (artificial intelligence) and NLP (natural language processing), the roots of all sentiment analysis software. He conducted an experiment using several tools and found they incorrectly classified content, even content from the tools’ own blogs. This is really chilling and you can read the entire paper here.

Freshminds conducted a similar experiment looking specifically at the ability of top tools like Radian6 (now owned by Salesforce), Brandwatch, and Sysomos. While the tools accurately predicted between 60 and 80% of utterances, when neutral utterances were removed (80% of the utterances) the accuracy dropped alarmingly.

It’s important to note the Freshminds study dates from 2010, however, Dr. Saba’s data is from just last year. Neither “trained” the tools. Training involves a laborious and time-consuming process of correcting predictions using manual coding of utterances so the tool learns to be better at predicting whether an utterance is positive, negative, or neutral. Because of this, many firms use sentiment analysis tools without training, making these experimental results applicable in many contexts.

Speech patterns

Even with extensive training, most tools experience difficulties analyzing utterances to detect sarcasm and other speech patterns that only emerge when utterances are analyzed holistically (in context rather than as isolated word patterns).

Why does sentiment analysis continue to grow?

Despite the problems highlighted in this post, the use of sentiment analysis tools continues to grow.

Why you might ask? For most firms, the answer lies in notions that only an automated tool can possibly analyze the vast amounts of data representing mentions of brands. We commonly hear this expressed as “BIG DATA”. And, certainly, an automated system is the only one capable of analyzing hundreds or thousands of mentions of your brand every day. Thus, even a tool that’s slightly accurate looks good when compared with ignoring all these mentions.

What we should be asking ourselves, however, is WHY we need to analyze ALL this data? Hence, requiring automated analysis. When the trade-off is huge inaccuracies and expending huge amounts of time to continually train your tool, is 100% analysis necessary to understand how consumers view your brand?

My answer is a resounding NO. And I offer this option. Wouldn’t it be better to analyze THOROUGHLY (manually) a random sample of utterances? In a recent presentation at the Text Analytics Summit in San Fransisco, I made exactly this case using data from several online sources. My analysis demonstrates the value of manual (computer-assisted analysis using Hyperresearch software, which is an analysis tool rather than a sentiment analysis tool) not only gives a clearer picture of sentiment but also provides a deeper understanding of how consumers view your brand. Companies can use this understanding to identify problems early, before they generate dissatisfaction, uncover opportunities, build advertising that resonates with consumers, and understand deeply the lives lived by their customers. This understanding helps the brand strategically do more than simple sentiment analysis.

Conclusion

You might find value in doing this type of analysis if you take the time to effectively train the software program and have the ability to delve more deeply into some results. For me, a better solution is to selectively analyze some utterances to gain a more nuanced vision of how consumers feel about your brand using some systematic process for identifying which utterances to analyze in a random way.

Need marketing help to support business growth?

We welcome the opportunity to show you how we can make your marketing SIZZLE with our data-driven, results-oriented marketing strategies.  Sign up for our FREE newsletter, get our FREE guide to creating an awesome website, or contact us for more information on hiring us.

Hausman and Associates, the publisher of MKT Maven, is a full-service marketing agency operating at the intersection of marketing and digital media. Check out our full range of services.