5
$\begingroup$

We need to make choices in our life and so we need to compare(=rate) things what is good what is bad. Question: are there some mathematical models which may capture features of some kind of voting/ratings systems ?

More precisely I want to have some model where there is some "ideal choice" (e.g. experts estimation of quality of question on MO) and result of real voting - which might be different. The model should help us to understand how to make ratings more close to "ideal choice". The possible application which I keep in mind is user's ratings of gadgets at amazon (or other sites), with the hope to improve it taking into account something like user's "reputation"...

Another application - current science journals play the role of "rating agencies" - papers published at "Annals" = stamp of great quality. Assume journals will disappear. Can we create a kind of crowd-sourcing rating which will be close to current journal based rating system ?

There are many votings systems - political elections, ratings of questions on stackoverflow, user's ratings of gadgets sold at amazon.com,... Of course, all of them have different features, however it might be that there are certain simple ideas which might be relevant for understanding how all these things work... To understand this: are there simple ideas or there are not? if there are - what are them ? is the gist of this question.

The question is very vague and so any comment is welcome.

$\endgroup$
1

3 Answers 3

13
$\begingroup$

Unfortunately, I don't expect that there's any clear-cut mathematical solution to this problem. The modelling issues are rather subtle, and different people will make very different assumptions in the framework they are using. I think there's a lot of value to exploring these possibilities, but it's likely to lead to a lot of competing, somewhat useful approaches, rather than a definitive or widely accepted solution.

Social choice theory deals with aggregating opinions from a lot of voters, and this is a huge field. (Donald Saari has written some great books on this topic.) The drawback is that it doesn't deal with the "ideal choice" aspect of your question, but rather just with which choice is most popular or representative in some way, which becomes subtle when there are more than two choices. Is it better for everyone to be lukewarm, or for some people to love a choice and others hate it?

There's been a lot of work in machine learning on learning from expert advice, where the experts may have different levels of expertise, which you don't know in advance and which can even change over time. Under certain assumptions, multiplicative weight algorithms and boosting are excellent ways to solve this problem, and they are quite simple and practical. (See http://www.cs.princeton.edu/~arora/pubs/MWsurvey.pdf and http://cseweb.ucsd.edu/~yfreund/papers/adaboost.pdf, for example.) This can be viewed as a reputation-based system, which rewards experts for good performance.

However, the assumptions are really critical. For example, these methods apply to cases where you get periodic objective feedback that can be used to judge the experts. In some cases, like journal ranking, you just don't: you learn something from measures like citation counts, but nobody thinks they are the best measure of quality, and you don't want the whole system to degenerate into a contest over who can best predict citation counts. On the other hand, a contest over who can predict repair rates for cars is not so crazy.

This is where the modelling gets tricky. People may not even agree on what they are trying to measure, let alone how to measure it. For example, which is more impressive: a relatively shallow paper that excites and inspires many people to do better work on the topic, or a deeper paper that plays a critical role in a smaller and arguably less important area? If we can't even settle that informally, it's hard to build a model.

There are also nontrivial issues of incentives. For example, the U.S. News & World Report college rankings are partly based on reputation surveys. Some university administrators have deliberately rated their institution highly and all others as inferior, to boost their rankings. (Clemson admitted this publicly; see http://www.insidehighered.com/news/2009/06/03/rankings.) The subfield of mechanism design within game theory addresses this issue of figuring out how to elicit information without giving anyone an incentive to lie. It can be done in some cases, but it's a hard problem, especially in cases like product reviews where, say, the manufacturer or a competitor may be hiring people behind the scenes to provide biased reviews. (I don't know of good statistics for this, but it is widely believed to be a serious problem.)

Overall, there are a lot of important ideas out there that are relevant to this topic, and I expect further progress. However, the modelling issue is a huge obstacle.

$\endgroup$
1
  • $\begingroup$ Do you have any further links to mechanism design literature that addresses rating mechanisms specifically? $\endgroup$
    – JeremyKun
    Aug 23, 2016 at 20:54
7
$\begingroup$

For the case of rating/voting systems there is a model which has been checked on the imdb movie data-base in the paper "Universality of movie rating distributions" (arXiv:0806.2305). You may also check this poster.

Disclaimer: This guy is actually my brother.

Moreover, there is some mathematical theory on consensus dynamics, that is, if a group of experts has to agree on some common opinion by an iterative process where they share their opinions and update their own ones in view of the others. Then one can try to analyze what factors influence the formation of a consensus. Some results (stated a bit vague) are:

  • Having some people with larger confidence for the others is helping to form consensus.
  • Increasing everybody's confidence for others does not always help.
  • Everybody should have some self-confidence.
  • Putting in more options (e.g. adding another degree of freedom to the opinion) help in forming a large "consensus cluster" but on the other hand it also increases the probability of having some extremists who will not reach this cluster.
$\endgroup$
4
$\begingroup$

If there are no incentive problems (i.e. if everyone is honestly seeking the truth) then Aumann's Agree-to-Disagree Theorem tells you that if the voters are allowed to revise their votes after seeing the outcome, they can't ultimately disagree (at least if there is some objectively correct answer). (And various results by people like Scott Aaronson suggest that the convergence to unanimity should be fast.)

So if we see persistent non-unanimity in these votes (which, obviously, we do), and if we want to understand what's going on, we've got to ask exactly why Aumann's Theorem doesn't apply, and that's harder than it looks (see various papers of Robin Hanson for why the most obvious solutions fail).

$\endgroup$
4
  • 2
    $\begingroup$ Aumann's theorem is beautiful and important, but it makes some very strong assumptions about ideal Bayesians and common knowledge, and my gut feeling is that they severely limit its applicability (so it should be viewed more as a disproof of these hypotheses). But I haven't read Hanson's papers and would love to understand this better. Where would you recommend starting? His paper "Are disagreements honest?" with Cowen looks appealing, but partly I chose it because it looks particularly readable, and I don't know how it fits in the big picture. $\endgroup$
    – Henry Cohn
    Mar 21, 2012 at 13:32
  • $\begingroup$ If you start with the Hanson/Cowen paper and follow up the references in its bibliography, you should be pretty much up to the state of the art. $\endgroup$ Mar 21, 2012 at 14:01
  • $\begingroup$ Great, thanks! I'll take a look at it. $\endgroup$
    – Henry Cohn
    Mar 21, 2012 at 14:08
  • $\begingroup$ I don't see the connection. If the posteriors are commonly known, they have to agree. But why should voting lead to commonly known posteriors? $\endgroup$ Mar 21, 2012 at 18:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.