Estimate rate of real correct/wrong from 4 answers quiz.

Question

I recently read that one in ten students think the first man on the moon was Buzz Lightyear, a Toy story cartoon. I'm not here to discuss the data in itself, rather, this reading got me into a problem I like just out of my interest for probability and statistics.

Suppose you have a quiz asking who was the first man on the moon, together with four answers:

1. Benjamin Franklin
2. Neil Armstrong
3. George Harrison
4. Buzz Lightyear

Now, if you give this test to a random number generator and you say they are students, the answers will be 25 % each. Clearly, you cannot declare 1 in 4 students believes Buzz Lightyear was the first man on the moon, and you also cannot declare 1 in 4 students knew the correct answer. It was just probability.

When you take such a test, you have these different cases:

1. people who know the correct answer and mark it
2. people who assume an incorrect answer as correct and mark it and
3. people who have no clue, and mark at random.

The results of the tests contain therefore a bias resulting from the "guessers", but hardly we can say that all people answering "Buzz Lightyear" really believed he was the first man on the moon. Most likely, some of them confused it with Buzz Aldrin and belong to case 2. Some others had no clue and threw a random choice. Same for the correct case: not all those who answered Armstrong really knew it. Some just guessed correctly.

Do you have any reference (or proposed solution) on this specific case to estimate the rate of really correct answers vs. random chance ?

Answer 1

As already pointed out, the model seems a bit too simple: In reality the students might be able to exclude one or two of the answers, or they might choose Buzz Lightyear, because they think it is a funny answer. But lets assume that a student either think he knows the answer, or uses a random number generator to decide what to answer (using uniform distribution). We want to find the fraction $a_0$ of students that don't think they know the answer, and the fraction $a_i$ of students, who think the answer is $i$. Now the fraction of students who answer $i$ is $p_i=a_i+\frac{a_0}{4}$. From the test, we are only able to estimate the numbers $p_i$ (unless we ask the students if they guessed) and in general it is not possible to determine say $a_4$ from these numbers. But we know that $p_i$ and $a_i$ are all positive, so we have: $0 \leq a_0 \leq 4\min(p_1,p_2,p_3,p_4)$ and $p_i-\min (p_1,p_2,p_3,p_4)\leq p_i-\frac{a_0}{4}=a_i\leq p_i$ for $ 1\leq i\leq 4$, so (under the assumptions) it is possible to estimate some upper and lower bounds on the number of students who think that Buzz Ligthyear was the first man on the moon: If 10% answered Buzz Lightyear and only 3% said Benjamin Franklin (or perhaps Neil Armstrong!) we could say that between 7% and 10% of all students think that Lightyear was the first man on the moon.

Answer 2

You are asking about using 4 or fewer observables to estimate the location of a point in a higher-dimensional space of models (ones that might explicitly account for guesses, deliberate jokes, confusion, or other routes to the answer other than believing that one of the given choices is correct). Clearly this is not possible in any strong sense but given some passable model of how answers are arrived at --- an algorithm imputed to respondents that has some parameters for the fraction of jokers, guessers etc --- one could get some idea of how big the bias could be. e.g., if 10 percent of the answerers are jokers and they split their votes between George Harrison and Buzz Lightyear this is already a majority of the BL votes.