# Entropy of Random Signal

How would you proof that for all equal variance (Continuous) Random Signals, a Gaussian one would have the largest Entropy. Or in other words, Given a Variance Gaussian PDF maximizes the Entropy.

It's easy to do with Lagrange Multipliers yet that you make the assumption the PDF is continuous and has a derive everywhere.

Is there a proof for any PDF? Not necessarily smooth?
Maybe using the Central Limit Theorem?

P.S.
Would someone add the "Entropy" Tag for this question.

-

Cover and Thomas's book is indeed the right place to learn about this.

The statement basically follows by convexity, in the form of Jensen's inequality. Here is the way it is usually presented:

Let $f$ be the probability density of a real random variable. Then the Shannon entropy is given by

$-\int f\log f dx$

You want to prove among all real random variables with finite Shannon entropy and variance equal to $1$, the Shannon entropy is maximized only for Gaussians.

Given two probability densities $f$ and $g$, since $\log$ is a concave function, Jensen's inequality tells us that

$\int f \log (g/f) dx < \log \int f(g/f) dx = \log \int g dx = 0$

Moreover, since $\log$ is strictly concave, equality holds if and only if $g = f$. If you now set $g$ equal to the probability density of a Gaussian with the same variance as $f$ and plug in the explicit formula for $g$, you get what you want.

-
Didn't get how is should work for not smooth functions. – Drazick Jan 12 '10 at 8:56
Nothing about this argument requires $f$ or $g$ to be smooth. – Mark Meckes Jan 13 '10 at 16:10
So I don't get it. Let's say g is a PDF of a random variable with finite Entropy and Variance = 1 not necessarily smooth (Has no derive at some points). f is a PDF of Gaussian random variable with Variance = 1. How does your proof work? I guess I'm missing something simple, sorry for not seeing it. Could you retry to explain it? Thanks. – Drazick Jan 14 '10 at 19:27
I don't understand what your problem with Deane's explanation is, unless you're missing that you should use $\log(g/f)=\log g−\log f$ to rewrite the inequality Deane wrote down. Then when you let $g$ be a Gaussian density (as Deane suggests; you switched the roles of $f$ and $g$), $\int f\log g dx$ is a simple linear combination of the mean and variance of $f$. The derivative of $f$ simply never enters the picture; $f$ doesn't even need to be continuous. – Mark Meckes Jan 15 '10 at 14:40
What I don't see is why if I set g and f otherway around it doesn't prove just the opposite? Where do we use the properties of f or g to prove the one with Gaussian Distribution has a bigger Entropy (Of course I was aware of the Log property). – Drazick Jan 16 '10 at 9:35

Ok first, the entropy you're talking about is the differential entropy $-\int f(x) \ln f(x) d\mu(x)$. The problem is that $\mu$ is Lebesgue measure. The set of continuous probability distributions is the set of distributions that have a density (i.e. radon-nikodym derivative) wrt Lebesgue measure. As such, if your distribution does NOT have such a density, then there isn't really a meaningful interpretation of the above quantity. Not only that, if your distribution had a density $f$ wrt to some other measure $\nu$, and you plugged $f$ into the above, you'd accidentally be computing the differential entropy of some completely different distribution $f d\mu$, which is a continuous distribution.

So to answer the question: if you want to deal with non-continuous distributions, you have to tweak your definition of differential entropy. if the only difference you make is to substitute in some well-behaved measure $\nu$, even if the derivation goes through the same, the distribution you get out will be wrt $\nu$, ie not the same as $\phi d\mu$ where $\phi$ is the density of the gaussian. (how much of that derivation you can re-use depends on the measure you choose.)

PS a good reference on this stuff is cover&thomas's information theory book, which has a derivation of gaussian being the max (differential) entropy (continuous) distribution with constant variance.

EDIT I misunderstood the question; I thought it was asking about entropy for distributions without a density wrt Lebesgue; all it is asking for is a proof without any conditions on the density. Deane Yang provides such a proof in his answer to the question.

-
Well, I'm coming from the EE world. I'm not familiar with some of the concepts you presented. Could you make it a bit clearer? Thanks. – Drazick Jan 12 '10 at 8:54
Sure. First let me make sure I understood your question: you are comfortable with the proof that, out of the set of all continuous distributions with variance 1, the one with maximum entropy is the gaussian distribution. Your question then is to drop the continuity constraint, i.e. find the max-entropy general distribution with variance 1. What I'm trying to say is that the question is ill posed because differential entropy is defined wrt continuous distributions--ie distributions with a density $f$ wrt regular integration (Lebesgue measure). continuing in another comment.. – Matus Telgarsky Jan 12 '10 at 10:29
..anyway there's no way to plug a non-continuous distribution into the standard definition of differential entropy. (as in, the standard definition is implicitly 'with respect to lebesgue measure'. you'd have to define another form for general distributions, but honestly such a concept doesn't make sense to me. perhaps someone else here knows of such generalizations.) Another approach is to use the lebesgue decomposition theorem and use differential entropy for the part that has a density wrt lebesgue measure, and something else for the other part.. – Matus Telgarsky Jan 12 '10 at 10:32
My preliminary thought was using the Central Limit Theorem. As adding 2 independent variables "grows" the Entropy. 2 steps are needed for that: 1. Make sure any variable with a certain Variance can be described by the sum of 2 other independent variables. 2. The max Entropy is a result of infinite number of such variables -> Gaussian Variable. Now all I need is to break this "Gaussian" into the same number of variables as the original variable. Their variance is equal hence the Entropy of the Gaussian is bigger. Is that make any sense? – Drazick Jan 13 '10 at 9:32
Drazick: the proof you suggest can be made to work, but is vastly more difficult than the Jensen's inequality argument in Deane's post below. – Mark Meckes Jan 13 '10 at 16:09

There is a another nice proof that is based on the fact that the entropy of x+y is at least the minimum of the individual entropies. Then one shows that by adding more and more terms in the summation, the entropy can only increase, and by the CLT, one get that this is a Gaussian. A scaling is also needed somewhere, please see any advanced text on IT for the details. The thomas-cover book is not enough here.

-