0
$\begingroup$

Problem: Suppose that $K$ different students are ranked based on $N$ different parameters (such as Physics marks, English marks, Biology marks, IQ etc). The rank under each parameter can be repetitive i.e. if two persons have the same IQ then they will have the same rank under the IQ parameter. I want to combine these $N$ individual ranks into a composite rank so we can find find the best overall student to the worst.

Known methods: There are several knows algorithms in literature for for combining individual ranks into a composite ranking. Most of these algorithms work by maximizing a function the $N$ correlations coefficient between the individual ranks under a parameter and the composite rank. Few algorithms maximize the sum of the squares of the individual correlations, other maximize the minima of these $N$ correlation coefficient etc.

I do not find the idea of maximizing a function of the correlation coefficient to create the composite rank to be logically convincing enough to make practical sense although it has statistical backing up. Is there a better way of coming up with a composite rank?

$\endgroup$
3
  • $\begingroup$ There are many different notions of "distance", and I don't see a canonical choice. Different distances will induce different rankings. Are you thinking of a specific distance function? Also, a different distribution of subjects might change the rankings between the students. E.g., if each "math" grade is replaced by an "algebra" and a "geometry" grade, the ranking of students who have bad math grades suddenly worsens. $\endgroup$
    – Goldstern
    Dec 4, 2012 at 12:49
  • $\begingroup$ "I thought the distance was more logically convincing argument than the one where we simply maximized correlation." What "logically convincing argument" did you have in mind? $\endgroup$ Dec 4, 2012 at 13:24
  • $\begingroup$ Your question is ill-posed since there is no "best ranking" until you define what "good" means to you. As pointed out below, the right keywords are "vote theory" and especially "Arrow's theorem". $\endgroup$ Dec 4, 2012 at 18:51

2 Answers 2

1
$\begingroup$

Think of the parameters as "voters" and the students as "candidates". Then what you have is an election, and you want to give a set of rules to tally votes and select a winner. However, Arrow's Theorem states that under certain natural conditions your rules will not be compatible with the preferences of all voters.

In other words, the "best student" is not well defined in general (unless the same person is best according to every criterion).

$\endgroup$
4
  • 1
    $\begingroup$ This seems unconvincing to me since I see no clear reason you'd want to require independence of irrelevant alternatives here, and plenty of good reason not to. $\endgroup$ Dec 4, 2012 at 15:51
  • 3
    $\begingroup$ The bigger problem, of course, is that the original poster hasn't given us a clue what properties he wants his ranking method to have. $\endgroup$ Dec 4, 2012 at 15:53
  • 1
    $\begingroup$ The independence of irrelevant alternatives makes a lot of sense to me. A composite ranking that says that student A is the best, but that if we withdraw student B, then C is the best, seems not very convincing to me. I would mitigate the conclusion of Rodrigo Pérez though: the Arrow theorem lead us to choose as ranking one of the given parameter ranking (or rather, given the possibility of draws, to use a lexicographic order). This is much less a problem than for a democratic vote. $\endgroup$ Dec 4, 2012 at 18:50
  • $\begingroup$ Benoit: If A is first in math and reading, and third in science, I might want to rank him ahead of B, who is second in all subjects. But if A is first in math and reading, and 2032nd in science, I might want to rank him behind B. This violates IIA. $\endgroup$ Dec 4, 2012 at 19:37
0
$\begingroup$

Maybe your alternative method is improved by considering the real frontier made of the envelope defined from the linear combination of the non-dominated of your points (persons) in the N dimensional space. The performance of each student would then be obtained from the distance not to the optimal theoretical point but to its projection over that real frontier (in fact it should be the ratio between the student point distance to the origin and the distance to the origin of its projection over the frontier). This way you would be ranking against real and comparable performance, and the best student in one discipline could never be considered the worst overall.

This is a ranking in the spirit of DEA (Data Envelopment Analysis), widely covered in the literature.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.