1
$\begingroup$

Let $I(X,Y)$ be the mutual information between two continuous random variables $X$ and $Y$.

We have $I(X,Y) = H(X)-H(X|Y)$, and setting $X=Y$ leads to $I(X,X) = H(X)-H(X|X)$. If $X$ was discrete, we'd have $H(X|X)=0$ and $I(X,X)=H(X)$, which is very intuitive. In a sense, $H(X|X)$ is the minimal entropy possible. However, differential entropy can be negative, so for continuous $X$ we get $H(X|X) = -\infty$, and thus $I(X,X) = +\infty$. That makes sense from a theoretical perspective, but any non-parametric mutual information estimator will give the wrong result. On the other hand, using $I(X,X)=H(X)$ in the continuous case underestimates the mutual information...

Improve this question
$\endgroup$
3
  • $\begingroup$ It might help you to know that the $\Delta-$ quantization $X^{\Delta}$ of a continuous random variable $X$ obeys $H(X^{\Delta})+\log \Delta \rightarrow h(X)$ as $\Delta\rightarrow 0.$ This clarifies the relationship between discrete and continuous entropy and shows that continuous entropy without specifying accuracy is not directly relevant to practice, e.g., estimation. If you quantize to $n$ bits, the quantization has entropy $h(X)+n$ and on average requires this much information to describe. $\endgroup$
    – kodlu
    Commented Jan 28, 2015 at 1:12
  • $\begingroup$ It looks like the last part of the above comment is missing $\endgroup$
    – Vincent
    Commented Jan 28, 2015 at 1:23
  • 1
    $\begingroup$ @kodlu: Thanks for your comment, but I'm not sure if you're right. Differential entropy is well-defined and discretization not necessary. If I sample from a multivariate Gaussian and use a non-parametric entropy estimator, the estimate converges to the expected theoretical value as the samples go to infinity. It's only the special case $H(x|x)$ that seems to be tricky for some reason. $\endgroup$
    – ASML
    Commented Jan 28, 2015 at 5:53

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .