Skip to main content
1 of 2
Royi
  • 115
  • 1
  • 8

My proof as I wrote few years ago on the Wikipedia Page.

Let $ g \left(x \right)$ be a Normal (Gaussian) Distribution Probability Density Function.
Since Differential Entropy is Translation Invariant and $ h \left( a X \right) = h \left( x \right) + \log \left( \left| a \right| \right) $ (See Properties of Differential Entropy) one could assume, without loss of generality, that $ g \left(x \right) $ is centered with its second moment to be 1:

$$ \int_{- \infty}^{\infty} g \left( x \right) x dx = 0 , \; \int_{- \infty}^{\infty} g \left( x \right){x}^{2}dx = 1 $$

Let $ f \left(x \right) $ be any arbitrary Probability Density Function with the same statistical properties for the first and second moment.

Applying on them the Kullback Leibler Divergence yields (Mind the Minus Sign):

$$\begin{align} -{D}_{KL} \left(f \left( x \right)\|g \left( x \right) \right) & = -\int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{f \left( x \right)}{g \left( x \right)} \right) dx \\ & = \int_{- \infty}^{\infty} f \left( x \right) \underset{\log \left( x \right) \leq x - 1}{\underbrace{\log \left( \frac{g \left( x \right)}{f \left( x \right)} \right)}} dx \\ & \leq \int_{- \infty}^{\infty} f \left( x \right) \left( \frac{g \left( x \right)}{f \left( x \right)}-1 \right) dx \\ & =\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0 \end{align} $$

Looking back at the term $ \int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx $ yields:

$$\begin{align} \int_{- \infty}^{\infty}f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx & = {h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx} } \\ & ={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \end{align} $$

Applying the inequality we showed previously we get:

$$ {h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0 $$

With the Equality holds only for $ g \left( x \right) = f \left( x \right) $.

Since this holds for any choice of $ f \left( x \right) $ (Namely any Probability Density Function with defined and finite first and second moment) it suggests that the Normal Distribution indeed maximizes the Differential Entropy form the set of PDF's with finite and defined 1st and 2nd moments.

Royi
  • 115
  • 1
  • 8