Differential Entropy of Random Signal

Question

Prove that the Normal (Gaussian) Distribution with a given Variance $ {\sigma}^{2} $ maximizes the Differential Entropy among all distributions with defined and finite 1st Moment and Variance which equals $ {\sigma}^{2} $.

Answer 1

Cover and Thomas's book is indeed the right place to learn about this.

The statement basically follows by convexity, in the form of Jensen's inequality. Here is the way it is usually presented:

Let $f$ be the probability density of a real random variable. Then the Shannon entropy is given by

$-\int f\log f dx$

You want to prove among all real random variables with finite Shannon entropy and variance equal to $1$, the Shannon entropy is maximized only for Gaussians.

Given two probability densities $f$ and $g$, since $\log$ is a concave function, Jensen's inequality tells us that

$\int f \log (g/f) dx < \log \int f(g/f) dx = \log \int g dx = 0$

Moreover, since $\log$ is strictly concave, equality holds if and only if $g = f$. If you now set $g$ equal to the probability density of a Gaussian with the same variance as $f$ and plug in the explicit formula for $g$, you get what you want.

Answer 2

Ok first, the entropy you're talking about is the differential entropy $-\int f(x) \ln f(x) d\mu(x)$. The problem is that $\mu$ is Lebesgue measure. The set of continuous probability distributions is the set of distributions that have a density (i.e. radon-nikodym derivative) wrt Lebesgue measure. As such, if your distribution does NOT have such a density, then there isn't really a meaningful interpretation of the above quantity. Not only that, if your distribution had a density $f$ wrt to some other measure $\nu$, and you plugged $f$ into the above, you'd accidentally be computing the differential entropy of some completely different distribution $f d\mu$, which is a continuous distribution.

So to answer the question: if you want to deal with non-continuous distributions, you have to tweak your definition of differential entropy. if the only difference you make is to substitute in some well-behaved measure $\nu$, even if the derivation goes through the same, the distribution you get out will be wrt $\nu$, ie not the same as $\phi d\mu$ where $\phi$ is the density of the gaussian. (how much of that derivation you can re-use depends on the measure you choose.)

PS a good reference on this stuff is cover&thomas's information theory book, which has a derivation of gaussian being the max (differential) entropy (continuous) distribution with constant variance.

EDIT I misunderstood the question; I thought it was asking about entropy for distributions without a density wrt Lebesgue; all it is asking for is a proof without any conditions on the density. Deane Yang provides such a proof in his answer to the question.

Answer 3

There is a another nice proof that is based on the fact that the entropy of x+y is at least the minimum of the individual entropies. Then one shows that by adding more and more terms in the summation, the entropy can only increase, and by the CLT, one get that this is a Gaussian. A scaling is also needed somewhere, please see any advanced text on IT for the details. The thomas-cover book is not enough here.

Answer 4

My proof as I wrote few years ago (Circa 2010) on the Wikipedia Page.

Let $ g \left(x \right)$ be a Normal (Gaussian) Distribution Probability Density Function.
Since Differential Entropy is Translation Invariant and $ h \left( a X \right) = h \left( x \right) + \log \left( \left| a \right| \right) $ (See Properties of Differential Entropy) one could assume, without loss of generality, that $ g \left(x \right) $ is centered with its second moment to be 1:

$$ \int_{- \infty}^{\infty} g \left( x \right) x dx = 0 , \; \int_{- \infty}^{\infty} g \left( x \right){x}^{2}dx = 1 $$

Let $ f \left(x \right) $ be any arbitrary Probability Density Function with the same statistical properties for the first and second moment.

Applying on them the Kullback Leibler Divergence yields (Mind the Minus Sign):

$$\begin{align} -{D}_{KL} \left(f \left( x \right)\|g \left( x \right) \right) & = -\int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{f \left( x \right)}{g \left( x \right)} \right) dx \\ & = \int_{- \infty}^{\infty} f \left( x \right) \underset{\log \left( x \right) \leq x - 1}{\underbrace{\log \left( \frac{g \left( x \right)}{f \left( x \right)} \right)}} dx \\ & \leq \int_{- \infty}^{\infty} f \left( x \right) \left( \frac{g \left( x \right)}{f \left( x \right)}-1 \right) dx \\ & =\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0 \end{align} $$

Looking back at the term $ \int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx $ yields:

$$\begin{align} \int_{- \infty}^{\infty}f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx & = {h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx} } \\ & ={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \end{align} $$

Applying the inequality we showed previously we get:

$$ {h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0 $$

With the Equality holds only for $ g \left( x \right) = f \left( x \right) $.

Since this holds for any choice of $ f \left( x \right) $ (Namely any Probability Density Function with defined and finite first and second moment) it suggests that the Normal Distribution indeed maximizes the Differential Entropy form the set of PDF's with finite and defined 1st and 2nd moments.