Impact of data collection method on NPS

Research has consistently shown that the method of data collection affects the responses given by the study participants. Among other factors, this is due to differences in the extent to which the participants feel under pressure to satisfice or to provide socially desirable responses (Holbrook et al, 2003).

We examined responses from a customer satisfaction study in which different data collection approaches were used to administer the same survey. The dual method had been adopted due to some sample accessibility challenges. The study had a sample of 1023 customers and of these 580 (57%) were interviewed using the computer aided telephonic method (CATI) and 443 (43%) were interviewed face to face.

One of the metrics in the study was the Net Promoter Score (NPS) in which customers who gave scores of 9 or 10 on their likelihood to recommend the service provider to others were categorized as Promoters.We found that 87% of the face to face respondents were in the Promoter category compared to 61% among the telephonic participants, and this was a statistically significant difference (p=0.01).

A follow up logistic regression model controlled for gender, age, race and region. The results showed that face to face respondents were 4 times more likely to be Promoters than those interviewed over the phone (OR = 4.2, 95% CI=3.0 - 5.9). This raised reliability and validity issues. We concluded that the face to face respondents had probably felt under more pressure to be "nice" on the service provider and therefore recommended that responses obtained using this method be discarded altogether. 

In any case, its better to assume happy customers are unhappy than to assume unhappy customers are happy.