I am a novice in an information theory and this is why my question is a bit naive))) Let us consider the following situation: Player A picks randomly a number from the set {0,1,2,3}. Player B must guess whether this number is even or odd. Player A follows the strategy of giving some partial information to the player B, namely if he picks number 0, then he says that this number is even but in three remaining cases payer A does not say anything (but player B does not know this strategy). Asuume that we want to decide what is the "information value" of messages of player A. There are at least three ways to do it:
- A very naive way is to say that player A gives 1 bit of information each 4-th time and hence the expectation of given information is 1/4=0.25 bit.
- A way to compute the information from the point of view of player B: without any information at all player B guesses right with probability 1/2. But with the help of the information from A he guesses rith with probability 3/4*1/2+1/4*1=5/8. Taking difference log(5/8)-log(1/2) we obtain log(5/4)=0.32... bit.
- If we use a definition that "information is a difference between full entropy and conditional entropy", then the answer is different. Full entropy of the test "even-odd" is 1 bit. Conditional entropy assuming that we know strategy and answers of A is
1/4*0 + 3/4*(log(3)/3+2/3*log(3/2))=1/4*log(27/4).
This gives the information value as the difference 1-1/4*log(27/4)=1/4*log(64/27)=0.31.... bit.
My question is: what is the most reasonable answer?
