Exploring misalignments between quantitative and qualitative data in NPS surveys

Introduction

The Net Promoter Score (NPS) is a metric used by businesses to gauge customer loyalty and satisfaction. It was introduced by Fred Reichheld in 2003 in his Harvard Business Review article "The One Number You Need to Grow." Since then, it has become a popular standard for measuring customer experience. NPS is based on a simple survey question: “On a scale of 0 to 10, how likely are you to recommend our product/service/company to a friend or colleague?” Customers respond with a number from 0 (not at all likely) to 10 (extremely likely).

Based on their scores, customers are then grouped into three categories:

• Promoters (score 9-10): Loyal enthusiasts who will keep buying and refer others, fuelling growth.

• Passives (score 7-8): Satisfied but unenthusiastic customers who are vulnerable to competitive offerings.

• Detractors (score 0-6): Unhappy customers who can damage your brand through negative word-of-mouth.

NPS is calculated by subtracting the percentage of detractors from the percentage of promoters, with the score ranging from -100 (if every customer is a detractor) to +100 (if every customer is a promoter).

While NPS is easy to understand and to track over time, allowing businesses to compare their customers’ satisfaction levels against competitors or industry standards, in doesn’t provide detailed feedback on why customers feel a certain way. It is for that reason that in market research surveys, the scaled NPS question is traditionally followed by an open-ended question asking respondents to elaborate on the numeric score they have given.

 Analysis objectives

Over the multitude of NPS surveys that we have run over the years, we have noticed that customers’ open-ended remarks explaining their numeric NPS scores do not always align with the Promoters versus Passives versus Detractors segmentation. We therefore sought to investigate this further by analysing open-ended remarks given to an NPS question by a sample of customers drawn from three large urban areas in South Africa, with the objective to assess the extent to which the qualitative sentiment aligned with the quantitative scoring.

Summary of findings

From the 794 customers who completed the survey, 293 (36.9%) fell in promoter category, 379 (47.7%) in the passives category, and 122 (15.4%) in the detractor category. The overall NPS was therefore 21.5. While specific benchmarks vary by industry and market, a general guideline is that an NPS of 0 to 50 is moderate, with room for improvement.

Sentiment analysis and coding of the open-ended remarks showed that, overall, 563 of the 794 (70.9%) of the respondents gave positive remarks, 202 (25.4%) gave neutral remarks, and only 29 (3.7%) had negative sentiments. Contrasted to the percentage of the respondents classified as detractors based on the earlier numeric scores, these numbers from the sentiment analysis were immediately suggestive of some mismatches between the quantitative and qualitative feedback.

Drilling deeper into the data, we found that among the promoters, 79.9% gave positive remarks, while 19.1% gave neutral remarks and the remaining 1.0% gave negative remarks. Within the passives, 74.9% gave positive remarks, 23.7% were neutral, and 1.3% negative. In the detractor segment, 36.9% gave positive remarks, 45.9% were neutral, and 17.2% gave negative remarks.

Discussion of the findings    

The promoters segment performed exactly as NPS theory predicts, with nearly 4 in 5 expressing genuinely positive sentiment in their verbatim feedback. The negligible negative sentiment (only 3 respondents) confirmed that the customers classified as promoters were authentic advocates who articulated clear, positive reasons for their high recommendation scores. The sentiment analysis showed a thin line between the promoters and passives and is largely in line with the NPS definition of passives as "satisfied but unenthusiastic" customers. The passives were confirmed as representing a conversion opportunity, with the small gap between their positive feedback and moderate scores probably indicating some missing "wow" factors what would make them "highly likely" to recommend.

Detractors showed the most heterogeneous sentiment profile and did not neatly align with the theoretical expectations of a predominantly negative customer segment, with only 17.2% of detractors expressing clearly negative sentiment. The plurality (45.9%) was neutral, often giving qualified or mixed responses. Most surprisingly, 36.9% expressed positive sentiment despite giving low recommendation scores.

Several factors could explain the observed discrepancy between the numeric scores and sentiment within the detractors group. It could be that the detractors liked the service provider overall but had specific pain points along the products and services value chain.

Another point is that customers perceive and use the 0-10 scale differently based on personal or cultural factors. For example, some customers may be reluctant to give a perfect 9 or 10 even if they are generally satisfied, resulting in a “Passive” score despite positive comments. A customer might give a moderately positive or neutral score but express a specific complaint or concern in their comments, leading to mixed signals. Conversely, a Detractor score (0-6) might come from a customer upset about a single issue, but their overall comment may still acknowledge positive aspects or loyalty. From yet another angle, customers may leave polite, positive comments out of courtesy, even if their numeric score reflects dissatisfaction. Or they might avoid harsh language but give a low score.

Overall, while the three NPS categories simplify customer sentiment, real opinions can be complex and multidimensional, making alignment tricky. For example, a customer may love product quality but dislike customer service, leading to conflicting score and comment signals.

Conclusion  

While NPS scores provide quick and easy to read dashboard type metrics, verbatim remarks and sentiment adds flesh to the bones. Unfortunately, there is an inclination amongst some end-users of NPS type surveys to construe apparent contradictions between numeric scores and qualitative feedback as evidence of poor data quality. For the same reason, some researchers feel compelled to edit or discard remarks that appear to contradict the numeric scoring, lest their work be deemed poor quality. In both instances, these are reactions to a false alarm.