2
$\begingroup$

We have a probability game, where we have $N$ number of events, each of which outcome can be $A,B$ or $C$. We do/will NOT know real probabilities afterwards: only the discrete outcome ($A, B$ or $C$) of each event.

Player 1 forecasts these events with certain probabilities (not only guess what is the outcome, but gives probabilities for each outcome option), and Player 2 as well with own probability estimates.

How we can know how accurate Player 1 and Player 2's predictions were (relation to reality) and how to measure the accuracy?

I have heard that one can use Akaike's information criterion to solve the problem. I was wondering another way but I need expert's opinion if this works:

I heard that one can start solving the problem by modeling the process by multinomic distribution and then take its Dirichlet's distribution. But how this leads to a solution?

Okay, I agree that one can write the solution like "Take some Dirichlet's distribution. Now use Akaike's information criterion". But I would like to know if this problem can be solved by using Dirichlet's distribution in some relative reasonable way so that you can't remove the distribution argument and the solution is still valid.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
1
$\begingroup$

I could be missing something here but I'd compute P[These events occur|Fortune teller 1 is telling the truth] and P[These events occur|Fortune teller 2 is telling the truth].

More explicitly Let N_a, N_b and N_c be the number of times outcomes A,B and C occured. Then P(This happened given fortune teller 1 told the truth)= (N choose a,b,c)[(P_1,a)^N_a][(P_1,b)^N_b][(P_1,c)^N_c] Where P_1,a is the probabiliy fortune teller one assigns to event a. (similiar for b and c).

Do the same computation for fortuneteller 2 and compare these 2. This tells us who assigned a higher probability to the outcome that came out. Suppose we might have some other criteria for "better" though (who is more often "close" for example but doubt it'll change the answer for "reasonable" definitions of "close").

Improve this answer
$\endgroup$
3
  • $\begingroup$ I was wondering how can this method works if one gives many correct predictions for easily guessed outcomes but another gives fewer correct predictions for some harder guessed predictions. $\endgroup$
    – Student
    Commented Nov 19, 2009 at 13:12
  • $\begingroup$ I might have been misreading here. Do the fortune tellers give a new distribution (probabilities of A B and C) before each event? Or do they just say here is the probability breakdown and it doesn't change. i.e. do we assume the results are IID? I assumed they where. $\endgroup$
    – Jonathan Kariv
    Commented Nov 19, 2009 at 14:11
  • $\begingroup$ I understood that they just say here is the probability breakdown and it doesn't change. $\endgroup$
    – Student
    Commented Nov 19, 2009 at 16:44
3
$\begingroup$

I was going to post almost this exact question today. Weird.

Anyway, I think we need some kind of model for the cost of getting a wrong probability before we can decide which fortune teller to use. Here's an approach that does that.

Suppose that we want to send a long N character message made up of symbols A, B and C over a noiseless channel but we don't know what the message is. Suppose a fortune teller tells us the probabilities of each symbol (possibly allowing for different (but still independent) probabilities for the same symbol in different positions in the message). We can use that information to construct a compression system for sending the message (supposing we can agree the compression system with the people at the other end before we know what the message is). We then get the actual message and send it. The cost is the number of bits required to send the message. The better the distribution fits the outcome, the fewer the bits we need. In the limit where the fortune teller gets every individual bit right, we don't have to actually send any message, it's already contained in the compression system description. (We need large N so that we can design a coding system that is close to the Shannon limit.)

This is the Kullback–Leibler divergence which is used to compare probability distributions. We're comparing the fortune teller's distribution with the probability distribution that is 1 for the actual outcome and 0 for everything else. Put another way it's the amount of extra information we learn on seeing the outcomes compared to what we knew based solely on the fortune teller's claims. We'd like this difference to be as small as possible.

Improve this answer
$\endgroup$
3
$\begingroup$

There is a problem in evaluating fortunteller or weather forcasters which is that when N is large they can demonstrate (almost) perfect fortcasting without knowing anything about the fortunes or weather, but only by calibrating their pervious answers. This is a theorem of Dean Foster and Rakesh Vohra The Annals of Statistics, 19, (1991), 1084 - 1090. See also the review paper by the same authors which discusses various related results "Regret in the On-line Decision Problem," Games and Economic Behavior, 1999, 7 - 36.

Improve this answer
$\endgroup$
0
$\begingroup$

Hi,

Jonathan, to answer your question: The probability distribution of each event, is unique (but now known). Also players will make predictions for each event. Events are 100% independent from the other events.

My method has been the following: 1. The general idea has been to generate "bets" between Player 1 and Player 2. For example, if Player 1 has higher probability on A (lets say 50%) than Player 2 (lets say 40%), Player 1 place a bet against Player 2 on event's outcome A. 2. In order to weight different variations of the probabilities (so that "bet" will be bigger then Player 1 predicts 90% rather than 50% against Player 2's 40%) and different probability of the outcome, I have used unique "bet size" for each even. I have followed "cash box" idea, where Players try to maximize the growth of their cashbox, when the "bet size" can be calculated with Kelly's formula, which is bet = (p1/p2-1) / (1/p2-1), where p1 and p2 are players 1 and player 2's probabilities for event A (and 1/p2 is the inverse probability of the event i.e. "odd"). 3. In simulation, I have substracted player 1's account by "bet" if event does NOT happen, and added by "bet" x (1-p2-1) if event occured 4. This is looped 1..N (number of events)

However, one probleme is how to know what is the reliability of the simulation. How about if we have just N=3, and in simplistic case lets assume we have "coin flipping" as event where in each event we have 50%/50% probability. If it happens, that a person predicts 70% for "heads" (stupid!!), but it happens so that 2 coins out of 3 are heads, the player 1 wins. But of course this is pure luck. But any way we can proof it?

Olli

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .