4
$\begingroup$

Let $I(X,Y):=H(X)+H(Y)-H(X,Y)$ be the mutual information of the joint probability distribution $p_{XY}$ (here $H(\cdot)$ is the Shannon entropy of its argument). I know that the mutual information is invariant IF $X'=\text{invertible_function}(X)$ and $Y'=\text{invertible_function}(Y)$, that is, it is invertible under a re-parametrization.

Is the converse true? I.e., is it true that $$I(X,Y) = I(X', Y')$$ implies that the marginals $X', Y'$ are invertible functions of $X$ and $Y$, respectively?

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

This is not true. Look at the following counterexample:

Suppose $X=X'$ and $Y'$ be a sufficient statistics of $Y$ with respect to $X$, that is we have a double Markov relation $X\to Y\to Y'$ and $X\to Y'\to Y$. The double Markov relation implies $I(X; Y')=I(X; Y)$. It also implies that either $X$ and $(Y, Y')$ are independent or $Y'$ is a deterministic function of $Y$. In non of these cases, $Y$ and $Y'$ are bijective.

There is another simple counterexample. Consider the trivial case that all $Y, Y'$ and $X$ are mutually independent; $I(X; Y)=I(X; Y')=0$, however, $Y$ and $Y'$ can be some arbitrary independent random variables.

Improve this answer
$\endgroup$
1
  • $\begingroup$ You're right, how in the world I didn't think about the super simple second example you gave... But the other implication should be true, that is, equality of mutual information under bijective local transformations. $\endgroup$
    – vsoftco
    Commented Jul 22, 2014 at 15:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .