Let $I(X,Y):=H(X)+H(Y)-H(X,Y)$ be the mutual information of the joint probability distribution $p_{XY}$ (here $H(\cdot)$ is the Shannon entropy of its argument). I know that the mutual information is invariant IF $X'=\text{invertible_function}(X)$ and $Y'=\text{invertible_function}(Y)$, that is, it is invertible under a re-parametrization.

Is the converse true? I.e., is it true that $$I(X,Y) = I(X', Y')$$ implies that the marginals $X', Y'$ are invertible functions of $X$ and $Y$, respectively?



1 Answer 1


This is not true. Look at the following counterexample:

Suppose $X=X'$ and $Y'$ be a sufficient statistics of $Y$ with respect to $X$, that is we have a double Markov relation $X\to Y\to Y'$ and $X\to Y'\to Y$. The double Markov relation implies $I(X; Y')=I(X; Y)$. It also implies that either $X$ and $(Y, Y')$ are independent or $Y'$ is a deterministic function of $Y$. In non of these cases, $Y$ and $Y'$ are bijective.

There is another simple counterexample. Consider the trivial case that all $Y, Y'$ and $X$ are mutually independent; $I(X; Y)=I(X; Y')=0$, however, $Y$ and $Y'$ can be some arbitrary independent random variables.

  • $\begingroup$ You're right, how in the world I didn't think about the super simple second example you gave... But the other implication should be true, that is, equality of mutual information under bijective local transformations. $\endgroup$
    – vsoftco
    Commented Jul 22, 2014 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.