Skip to main content
2 of 3
added 1408 characters in body
user75274
  • 231
  • 1
  • 3

A believe that a different way to go from differential entropy to discrete entropy is the following formula.

Let $h(X_1,\dots,X_n)$ be the differential entropy of some random $n$-tuple $X_1,\dots,X_n$ and $H(Y_1,\dots,Y_n)$ be the discrete entropy; we require that $Y_1,\dots,Y_n$ be real-valued. In other words, if $p$ is the joint density of $X_1,\dots,X_n$ then $$h= -\int p(x_1,\dots,x_n) \log p(x_1,\dots,x_n ) dx_1\dots dx_n$$ and $$H = -\sum_{\vec{v}} Prob( (Y_1,\dots,Y_n)=\vec{v}) \log \left(Prob( (Y_1,\dots,Y_n)=\vec{v})\right),$$ where the sum is taken over all vectors in $\mathbb{R}^n$.

The formula is: $$ H(Y_1,\dots,Y_n) = \lim_{t\to 0} h(Y_1 + tG_1,\dots,Y_n+tG_n) - n\log(t) $$ where $G_1,\dots,G_n$ are $n$ centered variance $1$ Gaussian random variables which are independent of each other and of $(Y_1,\dots,Y_n)$.

This formula explains why despite the fact that differential entropy changes when one applies a one-to-one function to $Y_1,\dots,Y_n$, the discrete entropy does not. Indeed if you set $Y_j' = F_j(Y_1,\dots,Y_n)$, then the RHS of the formula will involve a term which is (roughly) $\log \det \left(\left( \partial_i F_j\right)_{ij} \right)$, which disappears in the limit $t\to 0$ being negligible compared to $n\log t$.

(I am adding a few more details that we requested in the comments).

Take the case $n=1$ for simplicity. Then the law of $Y=Y_1$ has the form $\sum \alpha_j \delta_{x_j}$ with $\alpha_j\geq 0$, $\sum \alpha_j =1$, and $\delta_x$ denoting the point mass at $x$. Thus $H(Y)=-\sum \alpha_j \log \alpha_j$.

The law of $Y+tG$ is the convolution of the law of $Y$ and a Gaussian of variance $t^2$ centered at zero and so has density $$ p_t(x) = \frac{1}{\sqrt{2\pi}} \sum_j \alpha_j t^{-1} \exp( -(x-x_j)^2/t^2) $$

Let us now compute $h(Y+tG) = - \int p_t(x) \log p_t(x) dx$:

$$-\frac{1}{\sqrt{2\pi}} \int \sum_j \alpha_j \frac{e^{ - \frac{(x-x_j)^2}{t^2} }}{t} \sum_k \alpha_j \left[-\log t + \log \left(\sum \alpha_k e^{ - \frac{(x-x_k)^2}{t^2} }\right)\right]dx. $$

Noting that $\int \frac{1}{t} e^{ - \frac{(x-x_j)^2}{t^2} }=\sqrt{2\pi}$ the first term in the sum inside $[...]$ gives us $\sum \alpha_j \log t = \log t$. The second term is $$-\frac{1}{\sqrt{2\pi}} \int \sum_j \alpha_j \frac{e^{ - \frac{(x-x_j)^2}{t^2} }}{t} \log \left(\sum \alpha_k e^{ - \frac{(x-x_k)^2}{t^2} }\right)dx,$$ which because $-\frac{1}{\sqrt{2\pi}} \int \sum_j \alpha_j \frac{e^{ - \frac{(x-x_j)^2}{t^2} }}{t}$ converges weakly to $\sum \alpha_j \delta_{x_j}$ gives us precisely $-\sum \alpha_j \log \alpha_j$ (notice that $k=j$ gets forced as well).

Thus $h(Y+tG) -\log t \to H(Y)$.

The multi-dimensional case is essentially the same.

user75274
  • 231
  • 1
  • 3