4
$\begingroup$

What is the maximum entropy probability distribution if the support is a fixed interval (e.g. $[-1,1]$) with an already known variance?

If we know the support is a fixed interval, then the maximum entropy prob. distribution is the uniform distribution. But if we also add the variance constraint, I'm not sure what the answer is.

Thanks!

$\endgroup$
1

1 Answer 1

4
$\begingroup$

Denote the support by $[a,b]$ and the variance by $\sigma^2$. Repeating the variational calculus argument that shows the normal distribution maximizes the differential entropy (see here for instance), probability distributions of variance $\sigma^2$ and supported in $[a,b]$ that attain the largest possible entropy should have a density function of the form

$$ p(x)=\frac{e^{-\lambda(x-\mu)^2}}{\int_a^be^{-\lambda(t-\mu)^2}{\rm{d}}t}\quad (a\leq x\leq b)\tag{$\star$} $$ where $\lambda$ and $\mu$ satisfy

$$ \begin{cases} \int_a^bte^{-\lambda(t-\mu)^2}{\rm{d}}t=\mu\int_a^be^{-\lambda(t-\mu)^2}{\rm{d}}t,\\ \int_a^b(t-\mu)^2e^{-\lambda(t-\mu)^2}{\rm{d}}t=\sigma^2\int_a^be^{-\lambda(t-\mu)^2}{\rm{d}}t. \end{cases} $$ The first equation holds automatically if $\mu$ is at the center of the interval $(a,b)$. Assuming that $a,b$ are finite and the variational problem has a unique solution, then $\mu$ must be $\frac{a+b}{2}$: If $p:[a,b]\rightarrow(0,\infty)$ is the density function for the maximizer, then $x\in [a,b]\mapsto p(a+b-x)$ defines another probability distribution with the same support, variance and differential entropy. Then, the first equation above becomes irrelevant and one only needs to solve $$ \int_a^b\left(t-\frac{a+b}{2}\right)^2e^{-\lambda\left(t-\frac{a+b}{2}\right)^2}{\rm{d}}t=\sigma^2\int_a^be^{-\lambda\left(t-\frac{a+b}{2}\right)^2}{\rm{d}}t \tag{$\star\star$} $$ for $\lambda$.

Update) Following the comment by KConrad, a more general system of equations should be considered. Let's write the density function as

$$ p(x)=\frac{e^{-(\lambda x^2+\beta x)}}{\int_a^be^{-(\lambda x^2+\beta x)}{\rm{d}}t}\quad (a\leq x\leq b), \tag{$\star\star\star$} $$ In $(\star)$, it was implicitly assumed that $-\frac{\beta}{2\lambda}$ coincides with the mean $\mu$. Given the form $(\star\star\star)$ for the density function, the system of equations becomes

$$ \begin{cases} \int_a^bte^{-(\lambda t^2+\beta t)}{\rm{d}}t= \mu\int_a^be^{-(\lambda t^2+\beta t)}{\rm{d}}t,\\ \int_a^b(t-\mu)^2e^{-(\lambda t^2+\beta t)}{\rm{d}}t= \sigma^2\int_a^be^{-(\lambda t^2+\beta t)}{\rm{d}}t. \end{cases} $$

As argued before, assuming that the solution is unique, $p(x)$ must coincide with $p(a+b-x)$. When $p(x)$ is of the form $(\star\star\star)$, by comparing the coefficients of $x$ in the exponent from the numerator, $p(a+b-x)\equiv p(x)$ implies $-\frac{\beta}{2\lambda}=\frac{a+b}{2}$. Moreover, $\int_a^btp(t){\rm{d}}t=\int_a^btp(a+b-t){\rm{d}}t$ implies that the mean $\mu=\int_a^btp(t){\rm{d}}t$ is equal to $\frac{a+b}{2}$. Just like before, when $\mu=-\frac{\beta}{2\lambda}=\frac{a+b}{2}$, the first equation holds automatically because $\int_a^b\left(t-\frac{a+b}{2}\right)e^{-\lambda\left(t-\frac{a+b}{2}\right)^2}{\rm{d}}t=0$. So it only remains the second equation which would be same as $(\star\star)$, and should be solved for $\lambda$ to derive a density function of the form $$ p(x)=\frac{e^{-\lambda\left(x-\frac{a+b}{2}\right)^2}}{\int_a^be^{-\lambda\left(t-\frac{a+b}{2}\right)^2}{\rm{d}}t}. $$

$\endgroup$
4
  • 1
    $\begingroup$ In your equation to determine the mean, you are using $\mu$ in two different ways: the $\mu$ in the exponent need not be the mean $\mu$ outside the integral on the right side. On the interval $[0,1]$ with $p(x) = e^{-(x-1/3)^2}/\int_0^1 e^{-(x-1/3)^2}\,dx$, the mean $\int_0^1 xp(x)\,dx$ is around $.47407$, not $1/3$. $\endgroup$
    – KConrad
    Apr 14 at 14:39
  • 1
    $\begingroup$ So just the normal distribution conditioned on being within this interval? $\endgroup$ Apr 14 at 15:49
  • 1
    $\begingroup$ @KConrad I edited my answer. $\endgroup$
    – KhashF
    Apr 14 at 15:52
  • $\begingroup$ This may not work if the known variance $\sigma^2 > \frac{(b-a)^2}{12}$ unless you allow $\lambda$ to go negative; if you do, this would not be a truncated normal distribution $\endgroup$
    – Henry
    Apr 27 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.