Improving Sentiment Analysis: Star Rating Vs. Sentiment Index

In this article, Wonderflow’s Business Intelligence analysts shed more light and clarity on Star Rating and the Sentiment Index score, two very important metrics in sentiment analysis. This may help our readers, particularly current users, understand better why analyzing star rating alone, for example, is not enough to understand what customers want and need and to achieve product success.

 

What is the first thing you do when looking for a product on Amazon? You most likely check its star rating. 

If it’s too low, you just keep searching, but if it’s good enough and you are interested in buying, then you probably go and read some online reviews. There’s a reason you do that: the star rating alone doesn’t provide you with any information about the product and its features. It is just a number. 

For this same reason, companies are also interested in tracking not only what star rating their products get but also what customers say about them, positively or negatively.

This is why at Wonderflow, we provide two different metrics (among others) for our customers to measure and track the health of their products and brands: Star Rating and Sentiment Index.

But let’s take a step back and start by defining what these two sentiment analysis metrics are.

Star Rating is simply the average of all customer scores on a certain product they bought online. It goes from 1 to 5, where 1 star means that the customer is not happy at all with the product, while 5 stars mean they are very satisfied with the purchase. 

This Apple Watch sold on Amazon.com, for example, gets an impressive 4.8 as the Average Star Rating:

The Sentiment Index is a slightly more complex metric. Consider that each review can contain comments about different positive and negative aspects. Let’s look, for example, at the following review: 

Above, the customer is generally happy with the coffee machine. The quality of the coffee and design is poor, but he is dissatisfied with the steamer and instructions. 

So what the Sentiment Index tells us is what is the ratio between the amount of positive and negative words or comments. The Sentiment Index score plots the difference between the total positive and negative comments normalized in the interval , where +1 means that we only have positive comments and no negatives. 

-1 means that we have only negatives and no positives. 0 means that we have as many positive as negative comments.

This is how the Sentiment Index is calculated:

(Positive comments – negative comments) / (Positive comments + negative comments)

Analyzing these metrics together can provide a more comprehensive understanding of how a product or brand performs and the overall customer experience. As we initially said, the Average Star Rating is just a number, and customer evaluations are obviously subjective. 

So, for example, two customers can give a 4-star rating to a product but express a different opinion. The first one would report 1 positive aspect and 1 negative, while the other 3 positive and 1 negative aspect. So, we can have a more positive or negative sentiment for the same rating.

Let’s have a look again at real reviews like this one about a dishwasher:

Both customers gave a 4-star rating to the product, but we can easily appreciate from the reviews that their satisfaction with the product is quite different. While the second customer is enthusiastic about it, the first one has some areas to complain about and is just “all in all” happy about it. Not the same thing, right? 

So if we only look at the Star Rating, we would conclude that both customers are equally satisfied. Analyzing Sentiment Index, we would get a different picture and understand that one customer is less satisfied. Of course, collecting this kind of data for the hundreds and thousands of reviews that a product or brand might get can provide powerful information about its performance.

Here’s an example of two products, where one has a higher star rating but actually a lower Sentiment Index. If we look only at the star rating, we would conclude easily that product B is better than product A. Still, the Sentiment index tells another story: investigating the reasons behind this difference provides useful information on the overall performance of the product.

Massimo Paludet, Group Customer Care Director at the De’Longhi Group, is an expert user of the Wonderboard. He has confirmed how:

Star Rating and Sentiment Index can also be analyzed over time, providing useful additional information. At a general level, we expect to see a similar development of the two metrics because if customers are increasingly complaining about certain aspects of a product, we also expect its Star Rating to decline.

Or conversely, if people are saying more and more positive things, the Star Rating would increase, and that’s often the case. However, there might be situations where the two KPIs develop differently. 

Suppose we see, for example, that the Sentiment Index is increasing, but the Star Rating is staying flat. In that case, this might suggest that the topics that customers are talking about in a positive way don’t have such an impact on the rating they are giving. 

Let’s assume, in another example, that a new feature has been added to a product. Customers might add a positive comment in their reviews about that (“feature X works well,” “this is a nice feature to have”). 

This would increase the Sentiment Index since there’s one more positive topic to count. However, that feature might be just a nice-to-have; if it were not there, they would still give the same Star Rating.

Conversely, if we see a flat Star Rating but a declining Sentiment Index, this might be a red flag that the Star Rating may also start decreasing over time. The reason is there’s likely some aspect that customers are increasingly complaining about, which might soon affect their overall satisfaction.

Therefore, seeing discrepancies in the trendline of these two metrics would bring us to investigate further what customers are saying and understand if there are any issues we need to solve or aspects we need to promote more.

Nowadays, Star Rating is a widespread metric, the most adopted way of evaluating something online. On almost every website that sells products or services, you will find yellow stars that indicate the level of customer satisfaction (sometimes, we can even find the star rating of the website itself). 

However, what Star Rating doesn’t tell you is why a user is giving 5 stars or 1 star. Only the review left by the user can provide such a level of detail. That is why the Sentiment Index is so important. Looking at single metrics in isolation increases the risk of having only a partial view of the picture.

Again, Massimo Paludet underlines how:

If you’re interested in understanding more about how the natural language processing technique of sentiment analysis works, including how AI can help predict star ratings, talk with one of our VoC experts today to see a free demo.