Improving Sentiment Analysis: Star Rating Vs. Sentiment Index
Published on — Written by Wonderflow
In this article, Wonderflow’s Business Intelligence analysts shed more light and clarity on Star Rating and the Sentiment Index score, two very important metrics in sentiment analysis. This may help our readers, particularly current users, understand better why analyzing star rating alone, for example, is not enough to understand what customers want and need and to achieve product success.
What is the first thing you do when looking for a product on Amazon? You most likely check its star rating.
If it’s too low, you just keep searching, but if it’s good enough and you are interested in buying, then you probably go and read some online reviews. There’s a reason you do that: the star rating alone doesn’t provide you with any information about the product and its features. It is just a number.
For this same reason, companies are also interested in tracking not only what star rating their products get but also what customers say about them, positively or negatively.
This is why at Wonderflow, we provide two different metrics (among others) for our customers to measure and track the health of their products and brands: Star Rating and Sentiment Index.
But let’s take a step back and start by defining what these two sentiment analysis metrics are.
Star Rating is simply the average of all customer scores on a certain product they bought online. It goes from 1 to 5, where 1 star means that the customer is not happy at all with the product, while 5 stars mean they are very satisfied with the purchase.
This Apple Watch sold on Amazon.com, for example, gets an impressive 4.8 as the Average Star Rating:
The Sentiment Index is a slightly more complex metric. Consider that each review can contain comments about different positive and negative aspects. Let’s look, for example, at the following review:
Above, the customer is generally happy with the coffee machine. The quality of the coffee and design is poor, but he is dissatisfied with the steamer and instructions.
So what the Sentiment Index tells us is what is the ratio between the amount of positive and negative words or comments. The Sentiment Index score plots the difference between the total positive and negative comments normalized in the interval , where +1 means that we only have positive comments and no negatives.
-1 means that we have only negatives and no positives. 0 means that we have as many positive as negative comments.
This is how the Sentiment Index is calculated:
(Positive comments – negative comments) / (Positive comments + negative comments)
Analyzing these metrics together can provide a more comprehensive understanding of how a product or brand performs and the overall customer experience. As we initially said, the Average Star Rating is just a number, and customer evaluations are obviously subjective.
So, for example, two customers can give a 4-star rating to a product but express a different opinion. The first one would report 1 positive aspect and 1 negative, while the other 3 positive and 1 negative aspect. So, we can have a more positive or negative sentiment for the same rating.
Let’s have a look again at real reviews like this one about a dishwasher:
Both customers gave a 4-star rating to the product, but we can easily appreciate from the reviews that their satisfaction with the product is quite different. While the second customer is enthusiastic about it, the first one has some areas to complain about and is just “all in all” happy about it. Not the same thing, right?
So if we only look at the Star Rating, we would conclude that both customers are equally satisfied. Analyzing Sentiment Index, we would get a different picture and understand that one customer is less satisfied. Of course, collecting this kind of data for the hundreds and thousands of reviews that a product or brand might get can provide powerful information about its performance.
Here’s an example of two products, where one has a higher star rating but actually a lower Sentiment Index. If we look only at the star rating, we would conclude easily that product B is better than product A. Still, the Sentiment index tells another story: investigating the reasons behind this difference provides useful information on the overall performance of the product.
Massimo Paludet, Group Customer Care Director at the De’Longhi Group, is an expert user of the Wonderboard. He has confirmed how:
“Users tend to evaluate the relevance of the positive and negative aspects in relation to their individual expectations. In this sense, the Sentiment Index becomes a fundamental metric to turn aspects and considerations that are purely subjective into something objective, allowing companies to define action plans based on real priorities.”
Star Rating and Sentiment Index can also be analyzed over time, providing useful additional information. At a general level, we expect to see a similar development of the two metrics because if customers are increasingly complaining about certain aspects of a product, we also expect its Star Rating to decline.
Or conversely, if people are saying more and more positive things, the Star Rating would increase, and that’s often the case. However, there might be situations where the two KPIs develop differently.
Suppose we see, for example, that the Sentiment Index is increasing, but the Star Rating is staying flat. In that case, this might suggest that the topics that customers are talking about in a positive way don’t have such an impact on the rating they are giving.
Let’s assume, in another example, that a new feature has been added to a product. Customers might add a positive comment in their reviews about that (“feature X works well,” “this is a nice feature to have”).
This would increase the Sentiment Index since there’s one more positive topic to count. However, that feature might be just a nice-to-have; if it were not there, they would still give the same Star Rating.
Conversely, if we see a flat Star Rating but a declining Sentiment Index, this might be a red flag that the Star Rating may also start decreasing over time. The reason is there’s likely some aspect that customers are increasingly complaining about, which might soon affect their overall satisfaction.
Therefore, seeing discrepancies in the trendline of these two metrics would bring us to investigate further what customers are saying and understand if there are any issues we need to solve or aspects we need to promote more.
Nowadays, Star Rating is a widespread metric, the most adopted way of evaluating something online. On almost every website that sells products or services, you will find yellow stars that indicate the level of customer satisfaction (sometimes, we can even find the star rating of the website itself).
However, what Star Rating doesn’t tell you is why a user is giving 5 stars or 1 star. Only the review left by the user can provide such a level of detail. That is why the Sentiment Index is so important. Looking at single metrics in isolation increases the risk of having only a partial view of the picture.
Again, Massimo Paludet underlines how:
“The Sentiment Index provides a more trustworthy indication of the real degree of satisfaction of a product, being correlated to objective criteria (positive or negative), beyond star rating alone. It’s clear that we have to consider both KPIs, as companies tend to start working with the star rating and set targets on it. However, the Sentiment Index and the details related to it represent actionable feedback of huge value and relevance.”
If you’re interested in understanding more about how the natural language processing technique of sentiment analysis works, including how AI can help predict star ratings, talk with one of our VoC experts today to see a free demo.
Wonderflow empowers businesses with quick and impactful decision-making because it helps automate and deliver in-depth consumer and competitor insights. All within one place, results are simplified for professionals across any high-UGC organization, and department to access, understand, and share easily. Compared to hiring more analysts, Wonderflow’s AI eliminates the need for human-led setup and analysis, resulting in thousands of structured and unstructured reviews analyzed within a matter of weeks and with up to 50% or more accurate data. The system sources relevant private and public consumer feedback from over 200 channels, including emails, forums, call center logs, chat rooms, social media, and e-commerce. What’s most unique is that its AI is the first ever to help recommend personalized business actions and predict the impact of those actions on key outcomes. Wonderflow is leveraged by high-grade customers like Philips, DHL, Beko, Lavazza, Colgate-Palmolive, GSK, Delonghi, and more.
Other articles you might like:
Business · Dec 12, 2018
Review of the week: Polar watch Vantage V
You’ve probably heard the term sentiment analysis before but never really took much notice of how important it can be for your company. Wonderflow offers many tools that help brands discover the “why” behind customer feedback (ex. Why do customer dislike X feature?). This is because the language people are using can be mined to discover their true emotions and the intent in their sentences. With this information, you can gain a real understanding of…
Press · Apr 22, 2020
Wonderflow supports companies with free employee feedback analysis tool WonderWork
Wonderflow supports companies with free employee feedback analysis tool WonderWork WonderWork extracts actionable insights from employee feedback using artificial intelligence, aiming to help companies support new ways of working, as well as monitor and improve employee well-being. Amsterdam (NL) and Trento (IT), April 23, 2020 – Wonderflow, a leader in artificial…
Business · Mar 02, 2020
Video: What is the biggest limit of data science teams in an enterprise?
If we think about innovation today, the first thing that comes to our mind is probably related to information technology, complex software, and hardware. If we think about the word “innovation” inside a company, we are therefore pushed to think about the IT department, and we sometimes believe that all the new good…