Identifying winner of tournament (A,B testings) with none binary test result [closed]

Question

I'm trying to setup a tournament based on votes. Let say user vote for products A, B and C. Each user is presented all possible combinations of products in random order and he picks his preferred product. A user may also dismiss a test in which case each products gets half a point.

The points of each tests are summed, and divided by the number of users so that we end up with a matrix with a null diagonal and values between 0 and 1.

How do I deduce a score for each product from this matrix so that I can sort the products in decreasing order of preference ?

Answer 1

There is in fact research here and it gets at an interesting philosophical question: the interpretation of votes as expressions of preferences, versus as noisy observations of some underlying ground truth.

If one interprets votes as preferences, then one can use classic voting rules to aggregate these votes subject to certain fairness axioms. If one interprets votes as observations, then one can use statistical methods to aggregate the votes. For instance, suppose there is a true underlying ranking (perhaps $B \preceq C \preceq A$), and each pairwise vote you observe respects this ordering with probability $p$ and flips this ordering with probability $1-p$. Then one can ask for the MLE (maximum likelihood estimate) underlying ranking, given the votes.

The cool thing is that these approaches often coincide, for instance, the MLE described above is equivalent to the Kemeny voting rule.

This has been studied recently in the context of crowdsourcing as well. I think this is a very representative reference, and maybe can add others later.

• Better Human Computation Through Principled Voting. Mao, Chen, Procaccia, 2013.