1
$\begingroup$

Let $I(X,Y)$ be the mutual information between two continuous random variables $X$ and $Y$.

We have $I(X,Y) = H(X)-H(X|Y)$, and setting $X=Y$ leads to $I(X,X) = H(X)-H(X|X)$. If $X$ was discrete, we'd have $H(X|X)=0$ and $I(X,X)=H(X)$, which is very intuitive. In a sense, $H(X|X)$ is the minimal entropy possible. However, differential entropy can be negative, so for continuous $X$ we get $H(X|X) = -\infty$, and thus $I(X,X) = +\infty$. That makes sense from a theoretical perspective, but any non-parametric mutual information estimator will give the wrong result. On the other hand, using $I(X,X)=H(X)$ in the continuous case underestimates the mutual information...

$\endgroup$
3
  • $\begingroup$ It might help you to know that the $\Delta-$ quantization $X^{\Delta}$ of a continuous random variable $X$ obeys $H(X^{\Delta})+\log \Delta \rightarrow h(X)$ as $\Delta\rightarrow 0.$ This clarifies the relationship between discrete and continuous entropy and shows that continuous entropy without specifying accuracy is not directly relevant to practice, e.g., estimation. If you quantize to $n$ bits, the quantization has entropy $h(X)+n$ and on average requires this much information to describe. $\endgroup$
    – kodlu
    Jan 28, 2015 at 1:12
  • $\begingroup$ It looks like the last part of the above comment is missing $\endgroup$
    – Vincent
    Jan 28, 2015 at 1:23
  • 1
    $\begingroup$ @kodlu: Thanks for your comment, but I'm not sure if you're right. Differential entropy is well-defined and discretization not necessary. If I sample from a multivariate Gaussian and use a non-parametric entropy estimator, the estimate converges to the expected theoretical value as the samples go to infinity. It's only the special case $H(x|x)$ that seems to be tricky for some reason. $\endgroup$
    – ASML
    Jan 28, 2015 at 5:53

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.