How would you calculate the order of a list of reviews sorted by "Most Helpful" to "Least Helpful"?

Here's an example inspired by product reviews on Amazon:

Say a product has 8 total reviews and they are sorted by "Most Helpful" to "Least Helpful" based on the part that says "x of y people found this review helpful".

Here is how the reviews are sorted starting with "Most Helpful" and ending with "Least Helpful":

7 of 7
21 of 26
9 of 10
6 of 6
8 of 9
5 of 5
7 of 8
12 of 15

What equation do I need to use to calculate this sort order correctly? I thought I had it a few times but the "7 of 7" and "6 of 6" and "5 of 5" always throw me off. What am I missing?

  • 4
    $\begingroup$ By the time I found it this was closed, but I think there's a principled answer based on the assumption that each review i has probability p_i of being helpful. Assume a prior distribution on the values of p_i, apply Bayes rule to the observed data, and sort by the posterior probabilities. 7/7 should end up being better than 5/5. $\endgroup$ – David Eppstein Nov 18 '09 at 23:25
  • 3
    $\begingroup$ Why is this question considered "off topic"? It actually seems pretty interesting. $\endgroup$ – Darsh Ranjan Nov 19 '09 at 1:07
  • 3
    $\begingroup$ I disagree with closing this. There is an interesting question here: you have a bunch of {0,1} random variables, with different expected values, and you sample them different numbers of times. Your job is to reconstruct the relative ordering of the expectations. $\endgroup$ – David E Speyer Nov 19 '09 at 1:25
  • 4
    $\begingroup$ I concur. Just because the question was asked by an amateur (?) doesn't mean it's not an interesting question. $\endgroup$ – Jason Dyer Nov 19 '09 at 1:29
  • 5
    $\begingroup$ The answer by David Speyer pointing to "How not to sort by average ranking" is very interesting. I learnt something new, and there are valid discussions to be had about the assumptions that went into it and what alternatives might be used. It's also a really nice example of non-trivial mathematics meeting a very commonly asked real world problem. This should be encouraged, not discouraged. $\endgroup$ – Dan Piponi Nov 19 '09 at 1:54

See How not to sort by average ranking.


David Eppstein suggests a Bayesian method in his comment. One standard thing to do in this situation is to use a uniform prior. That is, before assessments of a review come in, its probability $p_i$ of being helpful is assumed to be uniformly distributed on [0, 1]. Upon receiving each assessment of the review, apply Bayes' theorem.

This sounds complicated, and it would be for an arbitrary prior distribution. But it turns out that with the uniform prior, the posterior distributions are all beta distributions. In particular, the expected value of $p_i$ after s positive assessments and n-s negative ones is (s+1)/(n+2). This is Laplace's rule of succession, and proofs of the facts I've mentioned can be found in that Wikipedia article. Then one would sort on the score (s+1)/(n+2).

The constants "1" and "2" come from the use of a uniform prior, and don't actually give the same results as the sample data you provide. But if you give a review that s out of n people have said to be helpful the score (s+3)/(n+6), then your reviews have scores

7 of 7: 10/13 = 0.769...

21 of 26: 24/32 = 0.75

9 of 10: 12/16 = 0.75

6 of 6: 9/12 = 0.75

8 of 9: 11/15 = 0.733

5 of 5: 8/11 = 0.727

7 of 8: 10/14 = 0.714

12 of 15: 15/21 = 0.714

This essentially amounts to sorting by the proportion of positive assessments of each review, except that each review starts with some "imaginary" assessments, three positive and three negative. (I don't claim that (3,6) is the only pair of constants that reproduce the order you give; they're just the first pair I found, and in fact (3k, 4k+2) works for any $k \ge 1$.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.