Almost two years ago I made a short blog post about how the Net Promoter Score (NPS), commonly used in business settings, is The Most Useless Metric of All. My reasons at the time is that it doesn't capture the reasons for a low score, it doesn't differentiate between subjective values in its scores, and it is mathematically incoherent (a three-value grade from an 11-point range of 0-10). Further, actual studies rank it last in effectiveness.

Recently, the author of the NPS, Fred Reichfield, has come around. Apparently now It's Not About the Score, but rather the score represents a "signal". This is a very far cry from the initial claims in the Harvard Business Review that it is "The One Number You Need to Grow". Of course, it is very difficult for anyone to admit they've made an error, and Reichfield is no exception to this. Instead of addressing what are real problems of the NPS method, he now tries to argue that people have gamified the scores, and that's the real problem. It would be great if as a general principle in business reasoning, people could just admit that their pet idea is flawed and build something better. That would be appreciated. Defending something that is clearly broken, even if it's your own idea, lacks intellectual humility, and is actually a bit embarrassing to watch.

Even as a signal, the NPS doesn't send a useful signal because the people being surveyed don't know what the signal means. In a scale where a 0 is equal to a 6, the scale is meaningless. There is, in fact, only three values in NPS (promoter, passive, detractor) and only one metric that it can possibly be testing: "How likely is it that you would recommend [company X] to a friend or colleague?"

Is that a useful question? Maybe for a generic good. It is far less useful for specialist goods. Do I recommend a three-day course in learning about job submission with Slurm for high-performance computing? Only to a few people that it would benefit. What score do I give? Maybe a 2, representing the number of people I would recommend it to? 3/11 is actually a lot in a quantitative sense, but that's the circles I mix with. Ah, but no; that makes me a detractor. And here we fall into the problem of subjective evaluation of the meaning of the scores.

The NPS doesn't send a useful signal also because the people receiving the survey have no idea what the signal means. "Wow, we're receiving a lot of positive promoters!", "Do you know why?", "Nope, but we must be doing something right. I wonder what it is?". It's like driving in the dark and congratulating your skills that you haven't gone off the edge of a cliff - yet. Who would do such a thing? The NPS, that's who.

To reiterate the post from two years ago, there are necessary changes needed to improve NPS. Firstly, if you're going to have a ranking method, use all the ranks! Also, 1-10 is a 10-point scale (which I suspect was the intention), not 0-10 - that's 11 (O is an index, people!). Secondly, ensure that there are qualitative values assigned to the quantitive values; 6/10 is not a detractor in a normal distribution - it's a neutral, leaning to positive. Specify how the quantitative values correlate with qualitative descriptions. Thirdly, actively seek out reasons for the rating provided. If you don't have that data all the signal will be is just that - a flashing light with no explanation. Without quantification and qualification, you simply cannot manage appropriately.

Finally, more questions! In managing customer loyalty, you will need to discover what they are being loyal to. It doesn't need to be overly long, just something that breaks down the experience that the customer can identify with. Customers may be lazy, but they're not that lazy. The benefit gained from a few questions provides much more insight than the loss of those customers who only answer one question: "A single item question is much less reliable and more volatile than a composite index. (Hill, Nigel; Roche, Greg; Allen, Rachel (2007). Customer Satisfaction: The Customer Experience through the Customer's Eyes). Yes, it is great to have people promoting your organisation or product. You know else you need? Knowledge of what that flashing light means.