Skip to main content
added 13 characters in body
Source Link
Royi
  • 115
  • 1
  • 8

My proof as I wrote few years ago (Circa 2010) on the Wikipedia Page.

Let $ g \left(x \right)$ be a Normal (Gaussian) Distribution Probability Density Function.
Since Differential Entropy is Translation Invariant and $ h \left( a X \right) = h \left( x \right) + \log \left( \left| a \right| \right) $ (See Properties of Differential Entropy) one could assume, without loss of generality, that $ g \left(x \right) $ is centered with its second moment to be 1:

$$ \int_{- \infty}^{\infty} g \left( x \right) x dx = 0 , \; \int_{- \infty}^{\infty} g \left( x \right){x}^{2}dx = 1 $$

Let $ f \left(x \right) $ be any arbitrary Probability Density Function with the same statistical properties for the first and second moment.

Applying on them the Kullback Leibler Divergence yields (Mind the Minus Sign):

$$\begin{align} -{D}_{KL} \left(f \left( x \right)\|g \left( x \right) \right) & = -\int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{f \left( x \right)}{g \left( x \right)} \right) dx \\ & = \int_{- \infty}^{\infty} f \left( x \right) \underset{\log \left( x \right) \leq x - 1}{\underbrace{\log \left( \frac{g \left( x \right)}{f \left( x \right)} \right)}} dx \\ & \leq \int_{- \infty}^{\infty} f \left( x \right) \left( \frac{g \left( x \right)}{f \left( x \right)}-1 \right) dx \\ & =\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0 \end{align} $$

Looking back at the term $ \int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx $ yields:

$$\begin{align} \int_{- \infty}^{\infty}f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx & = {h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx} } \\ & ={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \end{align} $$

Applying the inequality we showed previously we get:

$$ {h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0 $$

With the Equality holds only for $ g \left( x \right) = f \left( x \right) $.

Since this holds for any choice of $ f \left( x \right) $ (Namely any Probability Density Function with defined and finite first and second moment) it suggests that the Normal Distribution indeed maximizes the Differential Entropy form the set of PDF's with finite and defined 1st and 2nd moments.

My proof as I wrote few years ago on the Wikipedia Page.

Let $ g \left(x \right)$ be a Normal (Gaussian) Distribution Probability Density Function.
Since Differential Entropy is Translation Invariant and $ h \left( a X \right) = h \left( x \right) + \log \left( \left| a \right| \right) $ (See Properties of Differential Entropy) one could assume, without loss of generality, that $ g \left(x \right) $ is centered with its second moment to be 1:

$$ \int_{- \infty}^{\infty} g \left( x \right) x dx = 0 , \; \int_{- \infty}^{\infty} g \left( x \right){x}^{2}dx = 1 $$

Let $ f \left(x \right) $ be any arbitrary Probability Density Function with the same statistical properties for the first and second moment.

Applying on them the Kullback Leibler Divergence yields (Mind the Minus Sign):

$$\begin{align} -{D}_{KL} \left(f \left( x \right)\|g \left( x \right) \right) & = -\int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{f \left( x \right)}{g \left( x \right)} \right) dx \\ & = \int_{- \infty}^{\infty} f \left( x \right) \underset{\log \left( x \right) \leq x - 1}{\underbrace{\log \left( \frac{g \left( x \right)}{f \left( x \right)} \right)}} dx \\ & \leq \int_{- \infty}^{\infty} f \left( x \right) \left( \frac{g \left( x \right)}{f \left( x \right)}-1 \right) dx \\ & =\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0 \end{align} $$

Looking back at the term $ \int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx $ yields:

$$\begin{align} \int_{- \infty}^{\infty}f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx & = {h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx} } \\ & ={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \end{align} $$

Applying the inequality we showed previously we get:

$$ {h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0 $$

With the Equality holds only for $ g \left( x \right) = f \left( x \right) $.

Since this holds for any choice of $ f \left( x \right) $ (Namely any Probability Density Function with defined and finite first and second moment) it suggests that the Normal Distribution indeed maximizes the Differential Entropy form the set of PDF's with finite and defined 1st and 2nd moments.

My proof as I wrote few years ago (Circa 2010) on the Wikipedia Page.

Let $ g \left(x \right)$ be a Normal (Gaussian) Distribution Probability Density Function.
Since Differential Entropy is Translation Invariant and $ h \left( a X \right) = h \left( x \right) + \log \left( \left| a \right| \right) $ (See Properties of Differential Entropy) one could assume, without loss of generality, that $ g \left(x \right) $ is centered with its second moment to be 1:

$$ \int_{- \infty}^{\infty} g \left( x \right) x dx = 0 , \; \int_{- \infty}^{\infty} g \left( x \right){x}^{2}dx = 1 $$

Let $ f \left(x \right) $ be any arbitrary Probability Density Function with the same statistical properties for the first and second moment.

Applying on them the Kullback Leibler Divergence yields (Mind the Minus Sign):

$$\begin{align} -{D}_{KL} \left(f \left( x \right)\|g \left( x \right) \right) & = -\int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{f \left( x \right)}{g \left( x \right)} \right) dx \\ & = \int_{- \infty}^{\infty} f \left( x \right) \underset{\log \left( x \right) \leq x - 1}{\underbrace{\log \left( \frac{g \left( x \right)}{f \left( x \right)} \right)}} dx \\ & \leq \int_{- \infty}^{\infty} f \left( x \right) \left( \frac{g \left( x \right)}{f \left( x \right)}-1 \right) dx \\ & =\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0 \end{align} $$

Looking back at the term $ \int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx $ yields:

$$\begin{align} \int_{- \infty}^{\infty}f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx & = {h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx} } \\ & ={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \end{align} $$

Applying the inequality we showed previously we get:

$$ {h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0 $$

With the Equality holds only for $ g \left( x \right) = f \left( x \right) $.

Since this holds for any choice of $ f \left( x \right) $ (Namely any Probability Density Function with defined and finite first and second moment) it suggests that the Normal Distribution indeed maximizes the Differential Entropy form the set of PDF's with finite and defined 1st and 2nd moments.

Source Link
Royi
  • 115
  • 1
  • 8

My proof as I wrote few years ago on the Wikipedia Page.

Let $ g \left(x \right)$ be a Normal (Gaussian) Distribution Probability Density Function.
Since Differential Entropy is Translation Invariant and $ h \left( a X \right) = h \left( x \right) + \log \left( \left| a \right| \right) $ (See Properties of Differential Entropy) one could assume, without loss of generality, that $ g \left(x \right) $ is centered with its second moment to be 1:

$$ \int_{- \infty}^{\infty} g \left( x \right) x dx = 0 , \; \int_{- \infty}^{\infty} g \left( x \right){x}^{2}dx = 1 $$

Let $ f \left(x \right) $ be any arbitrary Probability Density Function with the same statistical properties for the first and second moment.

Applying on them the Kullback Leibler Divergence yields (Mind the Minus Sign):

$$\begin{align} -{D}_{KL} \left(f \left( x \right)\|g \left( x \right) \right) & = -\int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{f \left( x \right)}{g \left( x \right)} \right) dx \\ & = \int_{- \infty}^{\infty} f \left( x \right) \underset{\log \left( x \right) \leq x - 1}{\underbrace{\log \left( \frac{g \left( x \right)}{f \left( x \right)} \right)}} dx \\ & \leq \int_{- \infty}^{\infty} f \left( x \right) \left( \frac{g \left( x \right)}{f \left( x \right)}-1 \right) dx \\ & =\int_{- \infty}^{\infty}g \left(x \right) - f \left(x \right) dx=0 \end{align} $$

Looking back at the term $ \int_{- \infty}^{\infty} f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx $ yields:

$$\begin{align} \int_{- \infty}^{\infty}f \left( x \right) \log \left( \frac{g \left( x \right)}{f \left( x \right)} \right) dx & = {h}_{f \left(x \right)} \left(X \right) + \underset{=-{h}_{g \left(x \right)} \left(X \right)}{ \underbrace{\int_{- \infty}^{\infty}f \left(x \right) \log \left(g \left(x \right) \right)dx} } \\ & ={h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \end{align} $$

Applying the inequality we showed previously we get:

$$ {h}_{f \left(x \right)} \left(X \right) - {h}_{g \left(x \right)} \left(X \right) \leq 0 $$

With the Equality holds only for $ g \left( x \right) = f \left( x \right) $.

Since this holds for any choice of $ f \left( x \right) $ (Namely any Probability Density Function with defined and finite first and second moment) it suggests that the Normal Distribution indeed maximizes the Differential Entropy form the set of PDF's with finite and defined 1st and 2nd moments.