4
$\begingroup$

Here is a model which seems pretty close to my experience of writing multiple choice tests.

Let's view the answer $t$ to each question as a binary string in $S:=\{ 0,1 \}^k$, all equally likely. The scantron sheet offers $a$ answers per question, so the instructor must present the student with an $a$-element set $C$ of choices, one of which must be the true answer $t$. The student tries to solve the problem on his or her own, obtaining a string $s \in S$; each of the bits of $s$ has probability $p>1/2$ of matching the corresponding bit of $t$ and probability $1-p$ of being flipped. The student must then, as a function of $C$ and $s$, choose one of the elements of $C$.

A pure strategy for the instructor is a function from $S$ to $a$-element subsets of $S$, turning the true answer $t$ into the list of choices $C$. A mixed strategy for the instructor is a probability distribution on pure strategies.

The student knows the instructor's mixed strategy (the instructor has written many exams, which are now in the basements of fraternities) and know his probability of error $p$ (the student has taken practice exams). Therefore, upon seeing $C$ and computing $s$, the student performs a quick maximum likelihood analysis and returns the element of $C$ most likely to have lead to the pair $(C,s)$.

The instructor wants to minimize both Type I errors (students who have $s\neq t$ but answer correctly) and Type II errors (students who have $s =t$ but answer incorrectly). For concreteness sake, let's say the instructor wants to minimize the sum of these, although other weightings are certainly reasonable.

How well can the instructor do?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Perhaps "the student tries to solve the problem on his or her own"? $\endgroup$
    – Greg Martin
    Commented Jan 23, 2017 at 2:00
  • $\begingroup$ My plan was to write the student as male and the instructor as female, but it looks like the final draft left no pronouns referring to the instructor. Sure, I'll make your edit, that's easier than figuring out how to put some pronouns back. $\endgroup$
    – David E Speyer
    Commented Jan 23, 2017 at 2:01
  • $\begingroup$ And then from this MO question out pours the optimal strategy when guessing on a scantron!... I'm more interested in the student's strategy... $\endgroup$
    – user78249
    Commented Jan 23, 2017 at 5:10
  • $\begingroup$ Maybe I'm misreading you, but (as I read it) you wrote $t$ as the final list of $k$ answers given by the student, describing if they are right and wrong--isn't this sort of the wrong way of modeling it? Shouldn't you take the larger space of $S= \{1,2,3,...,a\}^k$ and assign a value $T \in S$ as the perfect score, and model the probability and likelihood off this? The more I read this question the more I feel it needs complexification and a more real world feel for the sheer cleverness of it. $\endgroup$
    – user78249
    Commented Jan 23, 2017 at 5:45
  • 1
    $\begingroup$ As an example: If the instructor offers the correct answer and a uniformly random sample of other answers, and the student chooses randomly among answers with the most bits matching their guess (I'm not sure if this is the maximum likelihood approach), with a=3, k=3, q=1-p, then errors are all type I and occur with probability (32ppq+4pqq)/21. $\endgroup$
    – user44143
    Commented Jan 23, 2017 at 17:05

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .