Estimate rate of real correct/wrong from 4 answers quiz.

I recently read that one in ten students think the first man on the moon was Buzz Lightyear, a Toy story cartoon. I'm not here to discuss the data in itself, rather, this reading got me into a problem I like just out of my interest for probability and statistics.

Suppose you have a quiz asking who was the first man on the moon, together with four answers:

1. Benjamin Franklin
2. Neil Armstrong
3. George Harrison
4. Buzz Lightyear

Now, if you give this test to a random number generator and you say they are students, the answers will be 25 % each. Clearly, you cannot declare 1 in 4 students believes Buzz Lightyear was the first man on the moon, and you also cannot declare 1 in 4 students knew the correct answer. It was just probability.

When you take such a test, you have these different cases:

1. people who know the correct answer and mark it
2. people who assume an incorrect answer as correct and mark it and
3. people who have no clue, and mark at random.

The results of the tests contain therefore a bias resulting from the "guessers", but hardly we can say that all people answering "Buzz Lightyear" really believed he was the first man on the moon. Most likely, some of them confused it with Buzz Aldrin and belong to case 2. Some others had no clue and threw a random choice. Same for the correct case: not all those who answered Armstrong really knew it. Some just guessed correctly.

Do you have any reference (or proposed solution) on this specific case to estimate the rate of really correct answers vs. random chance ?

-
Most philosophers would say that one can only `know' true things (or at least this is a highly standard position), and so you may want to adjust your terminology in item 2. It sounds peculiar to speak of people knowing an incorrect answer. –  Joel David Hamkins Mar 31 '10 at 1:54
@Joel : interesting point. changed. –  Stefano Borini Mar 31 '10 at 2:11
Possibly of interest: en.wikipedia.org/wiki/Dempster%E2%80%93Shafer_theory –  Steve Huntsman Mar 31 '10 at 2:57
Of course there is a refinement of 3., where people can narrow the correct answer down to say 2 choices and then guess randomly. Somehow we want to distinguish these people from the truly clueless people. –  Tony Huynh Mar 31 '10 at 3:10
Without some extra information or hypothesis, the evidence is consistent with everyone who answered "Buzz Lightyear" genuinely believing that, and also consistent with everybody making the "Buzz Aldrin" confusion. But it sounds as though you have some prior distribution in mind, which tells you that if someone answers "Buzz Lightyear" then the probability that they think the answer is a character in Toy Story is almost zero. I don't think you can do without that, so if you really want to understand this example you may need to supplement it with another experiment. –  gowers Apr 1 '10 at 13:49

As already pointed out, the model seems a bit too simple: In reality the students might be able to exclude one or two of the answers, or they might choose Buzz Lightyear, because they think it is a funny answer. But lets assume that a student either think he knows the answer, or uses a random number generator to decide what to answer (using uniform distribution). We want to find the fraction $a_0$ of students that don't think they know the answer, and the fraction $a_i$ of students, who think the answer is $i$. Now the fraction of students who answer $i$ is $p_i=a_i+\frac{a_0}{4}$. From the test, we are only able to estimate the numbers $p_i$ (unless we ask the students if they guessed) and in general it is not possible to determine say $a_4$ from these numbers. But we know that $p_i$ and $a_i$ are all positive, so we have: $0 \leq a_0 \leq 4\min(p_1,p_2,p_3,p_4)$ and $p_i-\min (p_1,p_2,p_3,p_4)\leq p_i-\frac{a_0}{4}=a_i\leq p_i$ for $1\leq i\leq 4$, so (under the assumptions) it is possible to estimate some upper and lower bounds on the number of students who think that Buzz Ligthyear was the first man on the moon: If 10% answered Buzz Lightyear and only 3% said Benjamin Franklin (or perhaps Neil Armstrong!) we could say that between 7% and 10% of all students think that Lightyear was the first man on the moon.