3
$\begingroup$

Why is Beta(1,1) the maximum entropy distribution over the bias of a coin expressed as a probability given that:

  • If we express the bias as odds (which is over the support $[0, \infty)$), then Beta-prime(1,1) is the corresponding distribution to Beta(1,1). Isn't the maximum entropy distribution over the positive reals the exponential distribution (which is not Beta-prime(1,1))?

  • If we express the bias in log odds (which is over the support of the reals), then the logistic distribution (with mean 0 and scale 1) is the corresponding distribution to Beta(1,1).

Beta(1,1) makes sense as maximum entropy because it's flat over its support. The other distributions are not flat. If we had chosen a different parametrization, we should clearly arrive at the corresponding distribution (not something else). How are the other two distributions the maximum entropy distributions over their support? There must be some other requirement that I'm missing. What is it?

Improve this question
$\endgroup$
4
  • $\begingroup$ You might want to cite some sources for the confusion. My guess is the underlying measure is not uniform over the real interval you are working with. $\endgroup$
    – John Jiang
    Commented Aug 29, 2010 at 3:46
  • $\begingroup$ The exponential distribution is the one that maximizes entropy SUBJECT TO a constraint on the expected value. $\endgroup$
    – Michael Hardy
    Commented Aug 29, 2010 at 5:00
  • $\begingroup$ Right, thanks. So, what's the constraint in this case? Surely, our expected odds is 1, but Beta-prime(1,1) is not the same distribution as Exponential(1). $\endgroup$
    – Neil
    Commented Aug 29, 2010 at 18:44
  • $\begingroup$ John, what do you mean cite sources for the confusion? $\endgroup$
    – Neil
    Commented Aug 29, 2010 at 18:46

2 Answers 2

Reset to default
2
$\begingroup$

I think there are two separate things going on here. One is the issue of a maximum entropy distribution. The other is of whether or not distributions are invariant under different parameterizations. Regarding the second matter, I think your statement "if we had chosen a different parameterization, we should clearly arrive at the corresponding distribution" is probably not quite right (I say probably because I may be interpreting you wrong). Only particular distributions have this property and sometimes are not probability distributions. See http://en.wikipedia.org/wiki/Jeffreys_prior if this is what you're interested in.

ps I'd have preferred to leave this as a comment, but can't yet I guess.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Here's my thinking: If you know nothing about a coin, then we're told that the maximum entropy distribution over the coin's bias is Beta(1,1). So, your answer to the question, "what is the probability that the coin's bias is less than 20%?" is 20%. If someone asks, "what is the probability that the coin's bias is less than odds 1:4?", you should also answer 20%. This is true for every probability, and so it stands to reason that we should be allowed to choose whatever parametrization we want. $\endgroup$
    – Neil
    Commented Aug 29, 2010 at 18:53
  • $\begingroup$ ...and we should get whatever pushforward probability measure would result given the mapping from the old sample space to the new one. (My wording might be off.) $\endgroup$
    – Neil
    Commented Aug 29, 2010 at 18:54
  • $\begingroup$ I think you're right. I need to read about the Jeffreys prior. I don't understand why his 64-year-old paper "An Invariant Form for the Prior Probability in Estimation Problems" is not freely available. $\endgroup$
    – Neil
    Commented Aug 29, 2010 at 19:04
  • 2
    $\begingroup$ This isn't an original source, but gives a clear look at the issue using your example: amstat.org/publications/jse/v12n2/zhu.pdf One basicaly requires Jacobians to cancel to get invariance. FWIW, the idea of reflecting "ignorance" with probability measures goes back all the way to Laplace and his principle of indifference. Max-ent and Jeffreys priors are two of the common approach. As you have discovered, they suggest different distributions. $\endgroup$
    – R Hahn
    Commented Aug 30, 2010 at 13:05
  • $\begingroup$ The max ent approach in some sense amounts to picking a sufficient statistic and using it to derive the distribution. In the case of the exponential, the sample space is the positive real line and the sufficient statistic is the sample mean. More generally, you can think of specifying sufficient statistics and an invariance property (formally a transition kernel) and deriving your distributions this way. This falls under the very pretty theory of "extremal families"; see eg Schervish's Theory of Statistics. $\endgroup$
    – R Hahn
    Commented Aug 30, 2010 at 13:08
2
$\begingroup$

Here's the Jeffreys paper: http://ifile.it/niok3wy/jeffreys-an_invariant_form-1946.pdf

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .