We have a probability game, where we have $N$ number of events, each of which outcome can be $A,B$ or $C$. We do/will NOT know real probabilities afterwards: only the discrete outcome ($A, B$ or $C$) of each event.
Player 1 forecasts these events with certain probabilities (not only guess what is the outcome, but gives probabilities for each outcome option), and Player 2 as well with own probability estimates.
How we can know how accurate Player 1 and Player 2's predictions were (relation to reality) and how to measure the accuracy?
I have heard that one can use Akaike's information criterion to solve the problem. I was wondering another way but I need expert's opinion if this works:
I heard that one can start solving the problem by modeling the process by multinomic distribution and then take its Dirichlet's distribution. But how this leads to a solution?
Okay, I agree that one can write the solution like "Take some Dirichlet's distribution. Now use Akaike's information criterion". But I would like to know if this problem can be solved by using Dirichlet's distribution in some relative reasonable way so that you can't remove the distribution argument and the solution is still valid.