Sorting Based on Rating

Question

I'm doing some machine learning stuff and I want to take some random samples and determine if a human agrees with the computer. To do this a user just votes up or down on a given item. Then I want to be able to sort by the items with the highest rating. I want to use something more complicated than simply up-down to get good results.

I've looked into the Wilson Interval Score and it seems like a decent solution, but I'm wondering if there are other alternatives.

I'm going to be using C# 4.0 if that matters but for now I'm strictly interested in the math.

Example below:

Lets suppose I have 3 items and multiple people have voted on them according to the table:

Item    Up    Down
1       6       1
2       60      11
3       100     40

In this example I would like Item 3 to be listed first, item 2 second and 3 third. This is a rough approximation of my expectations.

Item 3 has the most responses and highest relative approval. Item 2 has more responses than Item 1 despite having a lower percentage approval.

I'm trying to list the items in terms of some sort of relative metric and algrotithm without using something like percent approval or net score; something more complicated.

Answer 1

For the question as posed, the binomial model (Wilson Interval Score) is indeed a good approach.

However you should bear in mind that the votes are not independent events (as a binomial model supposes). You could do better by considering who is voting, and possibly also, who is viewing the list. That opens up a more complex range of models.

Collaborative filters are worth a look as a simple personalised ratings model. Beyond that, you get into designing a probability model to describe users and what they like, then fitting it to the data (e.g. by using the Newton-Raphson method to find the maximum likelihood).