13
$\begingroup$

What is the continuous probability distribution that maximizes entropy, given only the bounds of the random variable [a,b] and the mean mu of the probability distribution?

For example:

  • if a=0, b=1, and mu=0.5, it should return a U[0,1].

  • if a=10, b=20, and mu=20, it should return Dirac delta at x=20.

  • if a=0, b=1, and mu=0.8 it should return ... ?

I imagine the general solution will be based on the Beta distribution with some alpha and beta parameters expressed in terms of a, b, and mu, but I don't know.

Many thanks!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The answer is not going to be a Beta distribution. The maximum entropy distribution subject to the constraints that a particular finite set of random variables has specific values is going to be unique, and in your case the only expected value constraint you impose is the mean and Beta distributions are not determined by their mean values. $\endgroup$
    – KConrad
    Dec 18, 2012 at 1:26
  • 3
    $\begingroup$ If you want the Beta distribution for some parameter values to be the maximum entropy distribution on [0,1] subject to some constraints, then it is most "natural" to use the two weird constraints that $\int_0^1 p(x)\log x {\rm d}x$ and $\int_0^1 p(x)\log(1-x) {\rm d}x$ have specific values, where $p(x)$ is the unknown probability distribution. $\endgroup$
    – KConrad
    Dec 18, 2012 at 1:28

1 Answer 1

Reset to default
12
$\begingroup$

You can maximize the entropy using standard calculus of variations; you need to take into account the constraint that the probability distribution is properly normalized and that the mean is known, using Lagrange multipliers. You then find that the probability distribution is of the form:

$$p(x) = \frac{\alpha e^{\alpha x}} { (e^{\alpha b}-e^{\alpha a})}, x\in [a,b]$$

for $\alpha$ the unique solution to

$$\mu = \frac{ \int_a^b \alpha x e^{\alpha x} dx}{(e^{\alpha b} - e^{\alpha a})} = \frac{b e^{\alpha b} - a e^{\alpha a}}{(e^{\alpha b}-e^{\alpha a})} -\frac{1}{\alpha}$$

Improve this answer
$\endgroup$
12
  • 1
    $\begingroup$ Seems a little unlikely to find a distribution of that form supported on $[a,b]$. $\endgroup$
    – Anthony Quas
    Dec 18, 2012 at 1:32
  • 1
    $\begingroup$ Is that better? $\endgroup$
    – Will Sawin
    Dec 18, 2012 at 1:56
  • 1
    $\begingroup$ It may be helpful to think of this as a truncated exponential distribution...which direction it "faces" depends on if $\mu$ is bigger or smaller than the midpoint of the interval. When $\mu$ exactly equals the midpoint the max entropy distribution is clearly the uniform distribution on that interval. $\endgroup$
    – R Hahn
    Dec 18, 2012 at 2:16
  • 2
    $\begingroup$ I am writing up a derivation now that bypasses calculus of variations. The point is: how do you know for sure that the calculus of variations will give you the maximum entropy distribution, since a priori perhaps the maximum entropy distribution is not differentiable (but is continuous)? $\endgroup$
    – KConrad
    Dec 18, 2012 at 3:52
  • 3
    $\begingroup$ I posted a derivation as Theorem 5.1 at math.uconn.edu/~kconrad/blurbs/analysis/entropypost.pdf. $\endgroup$
    – KConrad
    Dec 18, 2012 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.