The Sad State of Sentiment Analysis

accuracy of sentiment analysisIf you’re an avid reader of this blog, you know I take a dim view of sentiment analysis, primarily because it’s a fairly shallow measure of your brand’s success. By shallow, I mean sentiment analysis only gives information about positive (or negative) mentions of your brand.

Problems with sentiment analysis

How negative?

While interesting, and potentially creating some value, sentiment analysis doesn’t tell you much about WHY folks feel that way about your brand. Results also don’t say much about HOW positively or negatively people feel about your brand — so real anger gets the same valence as mildly annoyed. Shallow.

Uncovering problems

Sentiment analysis is also a result measure. Not only doesn’t sentiment analysis tell you why people feel good or bad about your brand (although you can do a deep dive into actual conversations to discover this manually), but you’re finding out about the problem AFTER the emotion occurred, when it’s much harder to fix. It would be nice to know there was some problem so you could fix it BEFORE customers started sharing their negative emotions about your brand.

Collecting data

Existing sentiment analysis tools don’t collect EVERY mention of your brand. Most do a fairly good job of social media sites like Facebook and Twitter, but gathering information from blogs and news sites is much more limited. So, you don’t have complete information going into the analysis. That said, I don’t know of ANYTHING capable of collecting every online utterance.

As a manager, you must recognize that the absolute numbers of positive or negative utterances isn’t a true reflection of how folks feel about your brand. So, choosing to ignore a “few” negative comments is dangerous. Those few comments might represent 100’s or 1000’s of similar comments your sentiment analysis tool didn’t find.

Accuracy of sentiment analysis

While I’ve always questioned the accuracy of sentiment analysis results, a comment left on one of my earlier blog posts points to even less accuracy than I’d ever assumed.

Recent experiments suggest sentiment analysis data is LESS accurate than a coin toss (accuracy 50%). That’s really scary if your brand makes strategic decisions based on sentiment analysis.

If you’re a skeptic, like I am, here’s the proof:

Walid Saba is a professional in AI (artificial intelligence) and NLP (natural language processing), the roots of all sentiment analysis software. He conducted an experiment using several tools and found they incorrectly classified content, even content from the tools’ own blogs. This is really chilling and you can read the entire paper here.

Freshminds conducted a similar experiment looking specifically at the ability of top tools like Radian6, Brandwatch, and Sysomos. While the tools accurately predicted between 60 and 80% of utterances, when neutral utterances were removed (80% of the utterances) the accuracy dropped alarmingly.

It’s important to note the Freshminds study dates from 2010, however Dr. Saba’s data is from just last year. Neither “trained” the tools. Training involves a laborious and time-consuming process of correcting predictions using manual coding of utterances so the tool learns to be better at predicting whether an utterance is positive, negative, or neutral. Because of this, many firms use sentiment analysis tools without training, making these experimental results applicable in many contexts.

Speech patterns

Even with extensive training, most tools experience difficulties analyzing utterances to detect sarcasm and other speech patterns that only emerge when utterances are analyzed holistically (in context rather than as isolated word patterns).

Why sentiment analysis continues to grow?

Despite the problems highlighted in this post, the use of sentiment analysis tools continues to grow.

Why, you might ask? For most firms, the answer lies in notions that only an automated tool can possibly analyze the vast amounts of data representing mentions of brands. We commonly hear this expressed as “BIG DATA”. And, certainly, an automated system is the only one capable of analyzing 100’s or 1000’s of mentions of your brand every day. Thus, even a tool that’s slightly accurate looks good when compared with ignoring all these mentions.

What we should be asking ourselves, however, is WHY we need to analyze ALL this data? Hence, requiring automated analysis. When the trade-off is huge inaccuracies and expending huge amounts of time to continually train your tool, is 100% analysis necessary to understand how consumers view your brand?

My answer is a resounding NO. And I offer this option. Wouldn’t it be better to analyze THOROUGHLY (manually) a random sample of utterances? In a recent presentation at the Text Analytics Summit in San Fransisco, I made exactly this case using data from several online sources. My analysis demonstrates the value of manual (computer-assisted analysis using Hyperresearch software, which is an analysis tool rather than a sentiment analysis tool) not only gives a clearer picture of sentiment, but provides a deeper understanding of how consumers view your brand. Companies can use this understanding to identify problems early, before they generate dissatisfaction, uncover opportunities, build advertising that resonates with consumers, and understand deeply the lives lived by their customers. These understanding help the brand strategically must more than simple sentiment analysis.

Need Help

Whether you need a complete social media marketing strategy or some consulting to optimize your existing social media marketing, we can fill your digital marketing funnel or  developing a market information system that matches your needs. We can help you do your own social media marketing better or do it for you with our community managers, strategists, and account executives. You can request a FREE introductory meeting or sign up for my email newsletter to learn more about social media marketing.



You might also like:

Top 15 Social Media Analytics Software

sentiment analysisIt’s become increasingly important to use social media analytics to manage your social media marketing campaigns. These analytics allow you to optimize the ROI of your social media campaigns, manage them effectively, and monitor their performance.

As part of writing a social media analytics book, as well as wishing to provide the best analytics for my clients, I’ve tested or demoed a number of social media analytics tools. Some are free or have a nominal price, others are VERY pricy solutions. Some are easy to implement and others take a significant effort to use. For instance, the IBM social media analytics software requires JAVA snippets on each tracked page. Now, that might be feasible for sites with a large number of static pages, but those interested in assessing their blog will find putting unique JAVA snippets on each post cumbersome and awkward.

What do you need from your social media analytics software?

In working with clients (as well as managing my own social media marketing campaigns) there are 3 things I want my social media analytics software to do:

  1. Determine what’s working and what isn’t — do certain types of posts work best on certain platforms, what time works best for posting in each platform, which headlines work best, how do folks move through my website when I add a new landing page?????????
  2. How are folks moving through my conversion funnel? What problems do they encounter? Who are the folks who stop part way through the funnel?
  3. Who are my prospects and what do they think about me? What’s important to them? Who influences them? Where do they hang out? This includes sentiment analysis.

What your social media analytics software does for you

And, that’s another say these social media analytics solutions differ. Some are really good at answering 1 of my questions, for instance, SocialEars is really great at answering type 3 questions and Google Analytics is really good at answering Type 1 and Type 2 questions, but don’t do a thing to answer other questions. Some do a little to answer all 3 types of questions, but don’t do a great job with any.

If you’re an enterprise-level firm, maybe it doesn’t matter that you have to use several social media analytics tools — you can afford the cost and you have different folks working on different aspects of your social media marketing anyway, so each would only need answers to certain types of questions.

But, if you’re a small or mid-sized firm, like me, you can’t afford the time and expense involved in using multiple solutions. The best solution I’ve devised is combining free and paid solutions and bringing them together on a dashboard — like IBM’s Cognos, which is free.

15 social media analytics software

So, here’s my list (in random order) of what I think are the top social media analytics software tools. I’ve curated this list on Listly and you can add to the list as you wish be either adding new software tools or voting up the tools I’ve listed so they have a meaningful order. Enjoy

You might also like:

Social Media Analytics: Sentiment Analysis

sentiment analysisI talked a little about sentiment analysis earlier. Sentiment analysis mainly tracks the prevailing opinion of your brand and it’s most effective for larger brands, especially consumer brands. For smaller brands, there just aren’t enough mentions of your brand to make sentiment analysis very insightful. For small brands, scouring specific websites, such as Yelp, EOpinions, and other places likely provides more insights regarding prevailing opinions about your brand.

Sentiment analysis

Sentiment analysis combines natural language processing with artificial intelligence capability and text analytics to evaluate statements found across various social platforms to determine whether they are positive or negative with respect to a particular brand. A firm might track sentiment analysis over time to:

  • determine whether their actions improve or damage sentiment
  • track brand reputation
  • test how marketing efforts affect attitudes toward the brand
  • attitudes toward new products

For instance, maybe a user posts:

XYZ brand is bad ass!

Sentiment analysis must accurately categorize this as positive sentiment, despite the use of bad in the post. In other words, the tool needs to understand colloquial usage (natural language) to know that bad means good in this context. Artificial intelligence comes in when the tool is “trained”. So, let’s say the tool initially categorized the post as negative because of the word “bad” in the post. A human, as part of the training process, goes in and tells the system they were wrong – this is a positive statement. The tool “learns” that “bad ass” means good and doesn’t make that mistake again – at least not as often.

Now, I could spend several posts showing you how to calculate sentiment using advanced statistics and tuples (don’t ask – they give me a headache), but I assume you just want to use them, not create a tool to analyze sentiment. Hence, I’ll talk about tools already available as a SAAS (software as a service).

There are lots of tools out there for measuring and tracking sentiment. SAS and IBM both make excellent tools, but they’re expensive and a little unwieldy for the novice – designed more for business intelligence specialists. (marketing cloud), Trackur, Chatterbox, ViralHeat, and Radian6 all have sentiment analysis or include sentiment analysis as a piece of their social media automation. These tools are much more intuitive and user-friendly.

No matter how good, two problems exist with sentiment analysis: 1) training the system and 2) hearing. No tool is able to accurately score sentiment right out of the box and some are easier to train than others. That’s because sentiment is unstructured so the system has to be trained to determine if a particular statement is positive or negative. There’s also the issue of slang, misspelling, and weird sentence structure, which further complicates sentiment analysis. Training can be very time-consuming and tools vary in their ability to be trained, some can’t be trained at all. Across major tools, the industry average is about 65% accuracy – not so good in my opinion.

Hearing is the other problem with sentiment analysis. Some tools only listen to conversations in certain social media, although the ones I listed above listen across Facebook and Twitter and maybe a few other social platforms. Listening to blogs is a little more complicated because the posts are longer. Google Alerts can handle that, but I’ve not found them proficient at detected all conversations going on out there. Still, setting up an alert for your brands, key personnel, and some competitors makes sense. Google Alerts must be hand-scored or use a text analysis program, such as Nvivo or HyperRESEARCH.

A colleague interviewed for this book told the unfortunate story of hiring an intern to cull through the alerts and categorize them on a spreadsheet so the client could respond to unfavorable mentions. Unfortunately, this proved too big a task for the intern or Google. Thus, the client, a pharmaceutical company, was unaware of problems with one of their drugs until contacted by the FDA. Not exactly what the company wanted from their listening program.

Most work fairly similarly, so let’s talk about Trackur, since I have most experience with that platform. Above, you’ll see what one of the dashboards looks like.

Trackur lets you select keywords related to your brand. So, if I were using Trackur to monitor my business, I’d use keywords including my name and company name, if I had brand names, I’d use them, as well as names of key competitors, and industry names such as social media analytics. Trackur will track these keywords, calculated sentiment related to you and your brand, and chart changes in sentiment over time. Trackur has some nice features, in that it also gives you information about users mentioning your brand, thus combining some elements of influence analysis with sentiment analysis.

You might also like:

Social Media as Market Research

social media market researchSocial media is a fabulous market research tool! It allows you to step into everyone’s living room and hear the kinds of conversations they have around their dining room every night (well, assuming people still HAD conversations around their dining table).

Social media as market research

Businesses seem to always think about social media as a way to TALK to consumers, or get consumers to talk about them.  And, this message amplification is a great reason to use social media.  More people hear your message and they hear it from a “friend” — whether that’s you or someone else in their network.

Possibly more important for the success of the firm, is the potential to HEAR what consumers are saying.  You can hear what’s important for them, what they’re doing with their lives, who they identify with, what problems they encounter …

Knowing more about your target market helps you succeed in a number of ways:

1. Better advertising

Consumers buy products because they identify with the brand.  When you understand what consumers like and who they are, you can create ads that they identify with.  Ads that contain images of people who look like them, people who live the way they do, and images of people they admire.

2. New products

When you discover problems consumers encounter in their daily lives, you can create products customized to solve these problems.

The new liftgate on some SUV’s and Minivans is a good example of a product designed to solve a consumer problem.  When people use their cars, they often have their hands full with children, packages, sports equipment, or a briefcase and coffee cup.  You don’t have hands free to fumble around for your keys to push the liftgate button.  With the new liftgate, you don’t need to.  All you have to do is use your foot — and most people have a free foot!

3. Improved customer satisfaction

Despite the way if feels some days, few dissatisfied customers complain to the company about products.  But, they DO tell their friends.  And, with social media, consumers tell complete strangers on Twitter or Facebook, or Google+ when they encounter a problem.

If you’re listening, you can use these complaints to fix many problems.  Maybe the consumer didn’t understand how to use the product resulting in dissatisfaction.  You can help them understand or set up their product correctly by responding to their complaint in social media.

4. Modifications to existing products

Just as you can listen for dissatisfied customers and help them become more satisfied with the product, you can use customer complaints to make your brands better.  Maybe consumers post a desire to see the product in a different color.  Viola, you can get this into production fast.

Don’t just use comments to modify existing products, let consumers know you were LISTENING by telling them you made the changes.  This way, you’re forging better relationships with your customers, which leads to customer satisfaction, loyalty, and positive word of mouth.

California Tortilla does this well.  When customers complained about losing a menu item, the company brought it back for a limited time.

How to use social media for market research

1. Listening post

The first step is to collect data in a listening post.  I really like Radian6 for this, although it’s REALLY expensive.  Trackr is a good, less expensive alternative.

2. Analyze

The data generated by a listening post is called unstructured data, because it’s not numeric.  Analyzing unstructured data is much more challenging (but so are the rewards).

There’s software to help (and here are my 3 favorites), but dealing with unstructured data is more of an art than a science.  I’ll leave this topic for another post.

HyperRESEARCH is a product of ResearchWare.  I’ve used this extensively for qualitative research and find it very intuitive, easy to use and powerful.  While I’ve never used this for keyword based categorization, it should be able to handle this just fine.  One advantage of the software is it can handle audio and visual data (ie. YouTube and Flickr), however this data must be categorized manually (at least in the version I own).

nVIVO is another excellent qualitative analysis software program.  I’ve used an earlier version of this software with excellent results. Again, its intuitive and easy to use.  Its not as powerful as HyperRESEARCH, but still a good option.

SPSS text analysis software is a new addition to the SPSS family.  Since it shares an interface with familiar SPSS software its easy to use and subsequent data analysis can be accomplished using SPSS statistics after converting data to a numeric representation. A drawback is that this software doesn’t handle visual or auditory data.

Need help?

Just ask. We’re happy to put together a program to handle your needs whether it’s simple listening to monitor sentiment or a full-blown market research program (we can also do qualitative data research).  Visit our services page to learn more about what we can do to make your marketing SIZZLE or see a list of our satisfied customers.

You might also like:

Social Media Analytics: How’s Your Social Media Strategy Going?

einteractivesocial_media_analytics3Are you really kicking your social media marketing?

For too many folks the answer is:

I don’t know

And, that’s a problem.  Without good social media analytics and someone who knows how to interpret them, you just keep doing stuff and HOPE you’re getting some results.

What specific social media strategies are working? Which social networks work best? What types of content produce the best results on each social network? What time works best for posting on different social networks?

Who is reading your blog? Commenting on your Facebook posts? ReTweeting your Tweets?

How do you bring more traffic to your website?

Are you optimizing your social media strategy?

What’s the ROI of your social media strategy?

And, if you look at this infographic, you’ll see that more than half of all businesses can’t answer these questions.

Without good social media analytics, you’ll never answer these questions — and more

How to use social media analytics to make your social media marketing ROCK!

1. Determine appropriate metrics -

What are your KPIs (Key Performance Indicators)? In other words, what variables translate into ROI (Return on Investment) for the firm?

Of course the problem with marketing is that often it’s hard to translate social media marketing into tangible results — specifically sales.  But, combining social media analytics with market research, you can draw a line from social media results to market performance.

Another way of looking at KPIs is that they should fit with the goals of the firm.  And, these goals should be consistent with market performance, such as customer satisfaction improvements.

KPIs from your website or blog might include metrics such as #visitors, time on site, bounce rate, and views for specific content.  KPIs from your Facebook page may be measures of engagement.  Other KPIs might reflect your actions in social media, such as the average time between a comment on your Facebook page and your response.

2. Assess performance and optimize

Assessing performance is actually a multi-part activity.  First, you have to set up your social media strategy so it’s trackable.  That means creating landing pages, setting up codes within your site, create filters so you can see what visitors are doing using Google Analytics and Facebook Insights for Business.  On Twitter, you can use hashtags # to determine whether your social media strategy is working.

Without these coding tactics, it’s more difficult to use these tools to see what’s working.

3. Use tools to track social media interactions on sites you don’t own.

You’ll want to see how your brand is doing on social networks you don’t own.  And, you won’t have analytics for these sites.  That’s where tools like Radian6, Ubervu, and my personal favorite, Hootsuite Pro (get a FREE 30 trial of Hootsuite Pro). come in.  They track brand mentions across the internet — including on social media sites and websites.  Free tools such as Google Alerts and Social Mention can also help see what’s being said about your brand, although I don’t think these capture as much as the paid tools.  I use Google Alerts to supplement Hootsuite because, why not?

4. Interpret metrics

Social media analytics goes beyond just collecting the data, you have to interpret it! So, once you have the data, use it to determine what’s working and how to optimize your social media strategy.

  1. What do the metrics mean in terms of market performance variables such as satisfaction, loyalty, and endorsements?
  2. What kind of WOM (word of mouth) comes from your social media marketing efforts? Not only how many folks are talking about your brand, but what are they saying about it.
  3. How well are you doing connecting with your target market?

5. Optimize your social media strategy

What do your social media analytics say about your target audience?

Are you giving them what they want?

Is there something else your target audience needs?

How does your target audience live their lives?

How does your target audience respond to various types of content?

Need help creating or monitoring your social media strategy?

We’re here to help.  Hausman and Associates, the publisher of Hausman Marketing Letter, uses state of the art social media analytics and market research to optimize market performance on your brand online.  We’re happy to provide a proposal showing just how affordable a powerful social media policy can be.

Be sure to sign up for our email newsletter.  We’ve finished the first chapter of our new Social Media Analytics book and we’re looking for reviewers to give feedback.  Newsletter subscribers have first option for these reviewer spots — and get a FREE copy of the book for their efforts.

You might also like: