Skip to main content
1 of 3
user75274
  • 231
  • 1
  • 3

A believe that a different way to go from differential entropy to discrete entropy is the following formula.

Let $h(X_1,\dots,X_n)$ be the differential entropy of some random $n$-tuple $X_1,\dots,X_n$ and $H(Y_1,\dots,Y_n)$ be the discrete entropy; we require that $Y_1,\dots,Y_n$ be real-valued. In other words, if $p$ is the joint density of $X_1,\dots,X_n$ then $$h= -\int p(x_1,\dots,x_n) \log p(x_1,\dots,x_n ) dx_1\dots dx_n$$ and $$H = -\sum_{\vec{v}} Prob( (Y_1,\dots,Y_n)=\vec{v}) \log \left(Prob( (Y_1,\dots,Y_n)=\vec{v})\right),$$ where the sum is taken over all vectors in $\mathbb{R}^n$.

The formula is: $$ H(Y_1,\dots,Y_n) = \lim_{t\to 0} h(Y_1 + tG_1,\dots,Y_n+tG_n) - n\log(t) $$ where $G_1,\dots,G_n$ are $n$ centered variance $1$ Gaussian random variables which are independent of each other and of $(Y_1,\dots,Y_n)$.

This formula explains why despite the fact that differential entropy changes when one applies a one-to-one function to $Y_1,\dots,Y_n$, the discrete entropy does not. Indeed if you set $Y_j' = F_j(Y_1,\dots,Y_n)$, then the RHS of the formula will involve a term which is (roughly) $\log \det \left(\left( \partial_i F_j\right)_{ij} \right)$, which disappears in the limit $t\to 0$ being negligible compared to $n\log t$.

user75274
  • 231
  • 1
  • 3