Calculating the "Most Helpful" review - MathOverflow

Calculating the "Most Helpful" review

2009-11-18T22:13:17Z

How would you calculate the order of a list of reviews sorted by "Most Helpful" to "Least Helpful"?

Here's an example inspired by product reviews on Amazon:

Say a product has 8 total reviews and they are sorted by "Most Helpful" to "Least Helpful" based on the part that says "x of y people found this review helpful".

Here is how the reviews are sorted starting with "Most Helpful" and ending with "Least Helpful":

7 of 7
21 of 26
9 of 10
6 of 6
8 of 9
5 of 5
7 of 8
12 of 15

What equation do I need to use to calculate this sort order correctly? I thought I had it a few times but the "7 of 7" and "6 of 6" and "5 of 5" always throw me off. What am I missing?

Answer by David Speyer for Calculating the "Most Helpful" review

2009-11-18T22:23:25Z

See How not to sort by average ranking.

Answer by Michael Lugo for Calculating the "Most Helpful" review

2009-12-05T19:40:44Z

David Eppstein suggests a Bayesian method in his comment. One standard thing to do in this situation is to use a uniform prior. That is, before assessments of a review come in, its probability $p_i$ of being helpful is assumed to be uniformly distributed on [0, 1]. Upon receiving each assessment of the review, apply Bayes' theorem.

This sounds complicated, and it would be for an arbitrary prior distribution. But it turns out that with the uniform prior, the posterior distributions are all beta distributions. In particular, the expected value of $p_i$ after s positive assessments and n-s negative ones is (s+1)/(n+2). This is Laplace's rule of succession, and proofs of the facts I've mentioned can be found in that Wikipedia article. Then one would sort on the score (s+1)/(n+2).

The constants "1" and "2" come from the use of a uniform prior, and don't actually give the same results as the sample data you provide. But if you give a review that s out of n people have said to be helpful the score (s+3)/(n+6), then your reviews have scores

7 of 7: 10/13 = 0.769...

21 of 26: 24/32 = 0.75

9 of 10: 12/16 = 0.75

6 of 6: 9/12 = 0.75

8 of 9: 11/15 = 0.733

5 of 5: 8/11 = 0.727

7 of 8: 10/14 = 0.714

12 of 15: 15/21 = 0.714

This essentially amounts to sorting by the proportion of positive assessments of each review, except that each review starts with some "imaginary" assessments, three positive and three negative. (I don't claim that (3,6) is the only pair of constants that reproduce the order you give; they're just the first pair I found, and in fact (3k, 4k+2) works for any $k \ge 1$.)