most recent 30 from http://mathoverflow.net 2013-05-19T10:19:30Z http://mathoverflow.net/feeds/question/38217 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/38217/elo-rating-system-help-with-the-maths-around-number-of-matches Barry 2010-09-09T18:54:49Z 2010-09-14T01:45:52Z

<p>I'm creating a system that will allow people to rate images.</p> <p>My idea is to use an Elo Rating system (http://en.wikipedia.org/wiki/Elo_rating_system) for each image and then use crowdsourcing to have people say if an individual image is better than another i.e </p> <p>Is A better than B </p> <p>This will be used to updated the Elo rating of A and B, eventually I would end up ranking all the images from supposedly the best to worse.</p> <p>For this I have two questions</p> <ol> <li><p>Is this the correct use of Elo or should I be looking at another rating scheme.</p></li> <li><p>If the ELO rating is correct and I have 100 images how many "matches" do I need before I can confidently look at the ranking ?</p></li> </ol>

http://mathoverflow.net/questions/38217/elo-rating-system-help-with-the-maths-around-number-of-matches/38500#38500 Douglas Zare 2010-09-12T18:39:31Z 2010-09-12T18:39:31Z

<p>By the way, Elo ratings are named after Élő Árpád, so only the first letter should be capitalized. </p> <p>1) An implicit assumption of the Elo rating system is that if you know the true advantage of A over B, and of B over C, then you know the true advantage of A over C. I don't think that should hold for the preferences for images, so I don't think the Elo system will fit well. The consequence is that the ratings should change based on which matches you set up. If the Elo ratings were a good fit, then it should not matter much which opponent you choose for a particular image. In the Elo system, if A and B are equally matched, and C is preferred 2:1 over A, then the preference for C over B is precisely 2:1. I can imagine that this would not be the case at all, and that you could manipulate the rating of C by choosing whether it is paired against A rather than against B. </p> <p>Imagine a martial arts tournament. Perhaps an Elo rating would be appropriate if karate practitioners spar against each other, all trying to strike each other from a distance. However, very different skills would be used if they faced someone using judo who attempted to close to apply choke holds and throws, and there should be little reason to suppose that the variations between the karate students would predict how well they would do when facing someone using a completely different strategy. Back to images: Perhaps an Elo rating would work comparing images of different models of cars against each other for wallpaper, but would comparisons between cars predict how people compare an image of a car against an image of food or a person? Some people might almost always choose the image of food over a car as wallpaper, and others might do the reverse, which is only consistent with an Elo rating if the preferences between car images can't be strong.</p> <p>2) Different implementations of the Elo system use different parameters. However, in general, if the Elo system is appropriate, then an individual's rating in a large population behaves as a random walk in a potential well centered at the true rating. One consequence is that when a rating is close to the true rating, there is an exponential decay of the influence of perturbations. Another is that there is a stable distribution which is roughly normal about the true rating. I looked at these for Elo ratings in backgammon for a nonmathematical audience in <a href="http://www.bkgm.com/articles/Zare/Ratings.html" rel="nofollow">this article</a>, originally published in the online magazine GammonVillage.</p>

http://mathoverflow.net/questions/38217/elo-rating-system-help-with-the-maths-around-number-of-matches/38638#38638 sleepless in beantown 2010-09-14T01:45:52Z 2010-09-14T01:45:52Z

<p>My answers and my opinions.</p> <ol> <li><p>This is not the correct use of Elo. Elo requires something with a transitive property that holds over the ordinality of the elements of the set. A subjective rating such as ranking or ordering of images (based on what? beauty, attractiveness, hi-resolution, high percentage of high-frequency components, balanced color, etc.) will have great variability due to individual differences and preferences.</p></li> <li><p>N/A, since the Elo rating system will not help you in this case.</p></li> </ol> <p>Suggestion: consider using some sort of k-means segmentation or other clustering method to segment your user or subject population into groups with similar preferences. Then, in these specific subsets of your user population, it might be possible to average or coalesce the rankings provided.</p>