Prove that the Normal (Gaussian) Distribution with a given Variance $ {\sigma}^{2} $ maximizes the Differential Entropy among all distributions with defined and finite 1st Moment and Variance which equals $ {\sigma}^{2} $.


Cover and Thomas's book is indeed the right place to learn about this.

The statement basically follows by convexity, in the form of Jensen's inequality. Here is the way it is usually presented:

Let $f$ be the probability density of a real random variable. Then the Shannon entropy is given by

$-\int f\log f dx$

You want to prove among all real random variables with finite Shannon entropy and variance equal to $1$, the Shannon entropy is maximized only for Gaussians.

Given two probability densities $f$ and $g$, since $\log$ is a concave function, Jensen's inequality tells us that

$\int f \log (g/f) dx < \log \int f(g/f) dx = \log \int g dx = 0$

Moreover, since $\log$ is strictly concave, equality holds if and only if $g = f$. If you now set $g$ equal to the probability density of a Gaussian with the same variance as $f$ and plug in the explicit formula for $g$, you get what you want.

| cite | improve this answer | |
  • $\begingroup$ Nothing about this argument requires $f$ or $g$ to be smooth. $\endgroup$ – Mark Meckes Jan 13 '10 at 16:10
  • $\begingroup$ I don't understand what your problem with Deane's explanation is, unless you're missing that you should use $\log(g/f)=\log g−\log f$ to rewrite the inequality Deane wrote down. Then when you let $g$ be a Gaussian density (as Deane suggests; you switched the roles of $f$ and $g$), $\int f\log g dx$ is a simple linear combination of the mean and variance of $f$. The derivative of $f$ simply never enters the picture; $f$ doesn't even need to be continuous. $\endgroup$ – Mark Meckes Jan 15 '10 at 14:40
  • $\begingroup$ Your questions are reasonable, but please note that I only gave a sketch of the proof and there is no reason why you should believe that you understand the proof until you have worked out the details and actually tried doing some of the things you propose, like switching $f$ and $g$. You can also check at each step whether the derivative of $f$ ever enters into the proof. $\endgroup$ – Deane Yang Jan 16 '10 at 15:20
  • $\begingroup$ One more thing, The accurate way to prove it is using Log Sum Inequality (Which uses Jensen's Inequality). $\endgroup$ – Royi Jan 28 '10 at 17:23

Ok first, the entropy you're talking about is the differential entropy $-\int f(x) \ln f(x) d\mu(x)$. The problem is that $\mu$ is Lebesgue measure. The set of continuous probability distributions is the set of distributions that have a density (i.e. radon-nikodym derivative) wrt Lebesgue measure. As such, if your distribution does NOT have such a density, then there isn't really a meaningful interpretation of the above quantity. Not only that, if your distribution had a density $f$ wrt to some other measure $\nu$, and you plugged $f$ into the above, you'd accidentally be computing the differential entropy of some completely different distribution $f d\mu$, which is a continuous distribution.

So to answer the question: if you want to deal with non-continuous distributions, you have to tweak your definition of differential entropy. if the only difference you make is to substitute in some well-behaved measure $\nu$, even if the derivation goes through the same, the distribution you get out will be wrt $\nu$, ie not the same as $\phi d\mu$ where $\phi$ is the density of the gaussian. (how much of that derivation you can re-use depends on the measure you choose.)

PS a good reference on this stuff is cover&thomas's information theory book, which has a derivation of gaussian being the max (differential) entropy (continuous) distribution with constant variance.

EDIT I misunderstood the question; I thought it was asking about entropy for distributions without a density wrt Lebesgue; all it is asking for is a proof without any conditions on the density. Deane Yang provides such a proof in his answer to the question.

| cite | improve this answer | |
  • $\begingroup$ Well, I'm coming from the EE world. I'm not familiar with some of the concepts you presented. Could you make it a bit clearer? Thanks. $\endgroup$ – Royi Jan 12 '10 at 8:54
  • $\begingroup$ Sure. First let me make sure I understood your question: you are comfortable with the proof that, out of the set of all continuous distributions with variance 1, the one with maximum entropy is the gaussian distribution. Your question then is to drop the continuity constraint, i.e. find the max-entropy general distribution with variance 1. What I'm trying to say is that the question is ill posed because differential entropy is defined wrt continuous distributions--ie distributions with a density $f$ wrt regular integration (Lebesgue measure). continuing in another comment.. $\endgroup$ – Matus Telgarsky Jan 12 '10 at 10:29
  • $\begingroup$ ..anyway there's no way to plug a non-continuous distribution into the standard definition of differential entropy. (as in, the standard definition is implicitly 'with respect to lebesgue measure'. you'd have to define another form for general distributions, but honestly such a concept doesn't make sense to me. perhaps someone else here knows of such generalizations.) Another approach is to use the lebesgue decomposition theorem and use differential entropy for the part that has a density wrt lebesgue measure, and something else for the other part.. $\endgroup$ – Matus Telgarsky Jan 12 '10 at 10:32
  • $\begingroup$ My preliminary thought was using the Central Limit Theorem. As adding 2 independent variables "grows" the Entropy. 2 steps are needed for that: 1. Make sure any variable with a certain Variance can be described by the sum of 2 other independent variables. 2. The max Entropy is a result of infinite number of such variables -> Gaussian Variable. Now all I need is to break this "Gaussian" into the same number of variables as the original variable. Their variance is equal hence the Entropy of the Gaussian is bigger. Is that make any sense? $\endgroup$ – Royi Jan 13 '10 at 9:32
  • $\begingroup$ Drazick: the proof you suggest can be made to work, but is vastly more difficult than the Jensen's inequality argument in Deane's post below. $\endgroup$ – Mark Meckes Jan 13 '10 at 16:09

There is a another nice proof that is based on the fact that the entropy of x+y is at least the minimum of the individual entropies. Then one shows that by adding more and more terms in the summation, the entropy can only increase, and by the CLT, one get that this is a Gaussian. A scaling is also needed somewhere, please see any advanced text on IT for the details. The thomas-cover book is not enough here.

| cite | improve this answer | |

My proof as I wrote few years ago (Circa 2010) on the Wikipedia Page.

Let $ g \left(x \right)$ be a Normal (Gaussian) Distribution Probability Density Function.
Since Differential Entropy is Translation Invariant and $ h \left( a X \right) = h \left( x \right) + \log \left( \left| a \right| \right) $ (See Properties of Differential Entropy) one could assume, without loss of generality, that $ g \left(x \right) $ is centered with its second moment to be 1:

$$ \int_{- \infty}^{\infty} g \left( x \right) x dx = 0 , \; \int_{- \infty}^{\infty} g \left( x \right){x}^{2}dx = 1 $$

Let $ f \left(x \right) $ be any arbitrary Probability Density Function with the same statistical properties for the first and second moment.

Applying on them the Kullback Leibler Divergence yields (Mind the Minus Sign):

$$\begin{align} -{D}_{KL} \left(f \left( x \right)\|g \left( x \right) \right) & = -\int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{f \left( x \right)}{g \left( x \right)} \right) dx \\ & = \int_{- \infty}^{\infty} f \left( x \right) \underset{\log \left( x \right) \leq x - 1}{\underbrace{\log \left( \frac{g \left( x \right)}{f \left( x \right)} \right)}} dx \\ & \leq \int_{- \infty}^{\infty} f \left( x \right) \left( \frac{g \left( x \right)}{f \left( x \right)}-1 \right) dx \\ & =\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0 \end{align} $$

Looking back at the term $ \int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx $ yields:

$$\begin{align} \int_{- \infty}^{\infty}f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx & = {h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx} } \\ & ={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \end{align} $$

Applying the inequality we showed previously we get:

$$ {h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0 $$

With the Equality holds only for $ g \left( x \right) = f \left( x \right) $.

Since this holds for any choice of $ f \left( x \right) $ (Namely any Probability Density Function with defined and finite first and second moment) it suggests that the Normal Distribution indeed maximizes the Differential Entropy form the set of PDF's with finite and defined 1st and 2nd moments.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.