12
$\begingroup$

What is the continuous probability distribution that maximizes entropy, given only the bounds of the random variable [a,b] and the mean mu of the probability distribution?

For example:

  • if a=0, b=1, and mu=0.5, it should return a U[0,1].

  • if a=10, b=20, and mu=20, it should return Dirac delta at x=20.

  • if a=0, b=1, and mu=0.8 it should return ... ?

I imagine the general solution will be based on the Beta distribution with some alpha and beta parameters expressed in terms of a, b, and mu, but I don't know.

Many thanks!

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ The answer is not going to be a Beta distribution. The maximum entropy distribution subject to the constraints that a particular finite set of random variables has specific values is going to be unique, and in your case the only expected value constraint you impose is the mean and Beta distributions are not determined by their mean values. $\endgroup$ – KConrad Dec 18 '12 at 1:26
  • 2
    $\begingroup$ If you want the Beta distribution for some parameter values to be the maximum entropy distribution on [0,1] subject to some constraints, then it is most "natural" to use the two weird constraints that $\int_0^1 p(x)\log x {\rm d}x$ and $\int_0^1 p(x)\log(1-x) {\rm d}x$ have specific values, where $p(x)$ is the unknown probability distribution. $\endgroup$ – KConrad Dec 18 '12 at 1:28
11
$\begingroup$

You can maximize the entropy using standard calculus of variations; you need to take into account the constraint that the probability distribution is properly normalized and that the mean is known, using Lagrange multipliers. You then find that the probability distribution is of the form:

$$p(x) = \frac{\alpha e^{\alpha x}} { (e^{\alpha b}-e^{\alpha a})}, x\in [a,b]$$

for $\alpha$ the unique solution to

$$\mu = \frac{ \int_a^b \alpha x e^{\alpha x} dx}{(e^{\alpha b} - e^{\alpha a})} = \frac{b e^{\alpha b} - a e^{\alpha a}}{(e^{\alpha b}-e^{\alpha a})} -\frac{1}{\alpha}$$

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Seems a little unlikely to find a distribution of that form supported on $[a,b]$. $\endgroup$ – Anthony Quas Dec 18 '12 at 1:32
  • 1
    $\begingroup$ Is that better? $\endgroup$ – Will Sawin Dec 18 '12 at 1:56
  • 1
    $\begingroup$ It may be helpful to think of this as a truncated exponential distribution...which direction it "faces" depends on if $\mu$ is bigger or smaller than the midpoint of the interval. When $\mu$ exactly equals the midpoint the max entropy distribution is clearly the uniform distribution on that interval. $\endgroup$ – R Hahn Dec 18 '12 at 2:16
  • 2
    $\begingroup$ I am writing up a derivation now that bypasses calculus of variations. The point is: how do you know for sure that the calculus of variations will give you the maximum entropy distribution, since a priori perhaps the maximum entropy distribution is not differentiable (but is continuous)? $\endgroup$ – KConrad Dec 18 '12 at 3:52
  • 3
    $\begingroup$ I posted a derivation as Theorem 5.1 at math.uconn.edu/~kconrad/blurbs/analysis/entropypost.pdf. $\endgroup$ – KConrad Dec 18 '12 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.