I'm creating a system that will allow people to rate images.

My idea is to use an Elo Rating system (http://en.wikipedia.org/wiki/Elo_rating_system) for each image and then use crowdsourcing to have people say if an individual image is better than another i.e

Is A better than B

This will be used to updated the Elo rating of A and B, eventually I would end up ranking all the images from supposedly the best to worse.

For this I have two questions

  1. Is this the correct use of Elo or should I be looking at another rating scheme.

  2. If the ELO rating is correct and I have 100 images how many "matches" do I need before I can confidently look at the ranking ?

  • 1
    $\begingroup$ This is probably off topic... Have I been living under a rock and ELO is a well-known word now? $\endgroup$ Sep 9, 2010 at 18:58
  • 5
    $\begingroup$ Mariano, en.wikipedia.org/wiki/Electric_Light_Orchestra $\endgroup$
    – Will Jagy
    Sep 9, 2010 at 19:00
  • 3
    $\begingroup$ Here is a reference to the ELO Rating algorithm en.wikipedia.org/wiki/Elo_rating_system. It is used to rank "teams" in sports and from my research it seemed to fit into a maths type question $\endgroup$
    – Barry
    Sep 9, 2010 at 19:00
  • 2
    $\begingroup$ For what it's worth, the existence of the ELO system, if not its mathematical details, is well-known to chess players; for a bit of fun, you could add a "chess" tag, as any mathematician seriously interested in chess is quite likely to know the details of the ELO system. I suspect also it's not too hard for a good probabilist to answer this, so a "probability" tag might be useful. $\endgroup$
    – Zen Harper
    Sep 10, 2010 at 2:32
  • 3
    $\begingroup$ This is a typical machine learning problem. You are trying to learn user preferences. Scalar ratings like Elo cannot capture intransitive and clustered preferences. That's why sites like Amazon use context to make recommendations of the form "If you liked this product, you will probably also like products X, Y and Z". If you have more context, you can make even better predictions (e.g. if you ask the user to rate n images before making a prediction of an image they would like, you can treat it as a maximum entropy game of twenty questions over your database of past user preferences). $\endgroup$ Sep 13, 2010 at 4:00

2 Answers 2


By the way, Elo ratings are named after Élő Árpád, so only the first letter should be capitalized.

1) An implicit assumption of the Elo rating system is that if you know the true advantage of A over B, and of B over C, then you know the true advantage of A over C. I don't think that should hold for the preferences for images, so I don't think the Elo system will fit well. The consequence is that the ratings should change based on which matches you set up. If the Elo ratings were a good fit, then it should not matter much which opponent you choose for a particular image. In the Elo system, if A and B are equally matched, and C is preferred 2:1 over A, then the preference for C over B is precisely 2:1. I can imagine that this would not be the case at all, and that you could manipulate the rating of C by choosing whether it is paired against A rather than against B.

Imagine a martial arts tournament. Perhaps an Elo rating would be appropriate if karate practitioners spar against each other, all trying to strike each other from a distance. However, very different skills would be used if they faced someone using judo who attempted to close to apply choke holds and throws, and there should be little reason to suppose that the variations between the karate students would predict how well they would do when facing someone using a completely different strategy. Back to images: Perhaps an Elo rating would work comparing images of different models of cars against each other for wallpaper, but would comparisons between cars predict how people compare an image of a car against an image of food or a person? Some people might almost always choose the image of food over a car as wallpaper, and others might do the reverse, which is only consistent with an Elo rating if the preferences between car images can't be strong.

2) Different implementations of the Elo system use different parameters. However, in general, if the Elo system is appropriate, then an individual's rating in a large population behaves as a random walk in a potential well centered at the true rating. One consequence is that when a rating is close to the true rating, there is an exponential decay of the influence of perturbations. Another is that there is a stable distribution which is roughly normal about the true rating. I looked at these for Elo ratings in backgammon for a nonmathematical audience in this article, originally published in the online magazine GammonVillage.


My answers and my opinions.

  1. This is not the correct use of Elo. Elo requires something with a transitive property that holds over the ordinality of the elements of the set. A subjective rating such as ranking or ordering of images (based on what? beauty, attractiveness, hi-resolution, high percentage of high-frequency components, balanced color, etc.) will have great variability due to individual differences and preferences.

  2. N/A, since the Elo rating system will not help you in this case.

Suggestion: consider using some sort of k-means segmentation or other clustering method to segment your user or subject population into groups with similar preferences. Then, in these specific subsets of your user population, it might be possible to average or coalesce the rankings provided.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.