Invariance of mutual information under injective functions

Question

Let $X\colon \Omega\to\mathcal X$ and $Y\colon \Omega\to \mathcal Y$ be two random variables. In M.S. Pinkser's Information and information stability of random variables and processes mutual information is defined as:

$$I(X; Y) = \sup \sum_i P_{(X,Y)}(E_i) \log \frac{ P_{(X, Y)}(E_i)}{ P_{X\times Y}(E_i),} ,$$ where the supremum is taken over all finite partitions $\{E_1, \dotsc, E_n\}$ of $\mathcal X\times\mathcal Y$ and one uses the usual conventions $0\cdot \log 0/a = a\log a/0 = 0$ for $a\ge 0$.

In particular, if $P_{(X, Y)}$ is not absolutely continuous w.r.t. $P_{X\times Y} = P_X\otimes P_Y$, then $I(X; Y) = \infty$. In the other case, mutual information becomes the well-known formula involving the Radon–Nikodym derivative:

$$I(X; Y) = \int_{\mathcal X\times \mathcal Y} \log \left( \frac{ \mathrm{d}P_{X,Y} }{ \mathrm{d}(P_X\otimes P_Y) } \right)\, \mathrm{d}P_{X,Y}$$ and is finite.

One can prove that if $f\colon \mathcal X\to \mathcal X'$ and $g\colon \mathcal Y\to \mathcal Y'$ are measurable bijective mapping with measurable inverses, then $I(X; Y) = I(f(X); g(Y))$.

I wonder if this assumption can be weakened in the smooth case:

Assume that $\mathcal X = \mathbb R^m$ and $\mathcal Y = \mathbb R^n$ and $I(X; Y)$ is finite. Let $f\colon \mathbb R^m\to \mathbb R^{m+k}$ and $g\colon \mathbb R^n\to \mathbb R^{n+l}$ be smooth embeddings.

Is this true that $I(f(X); g(Y))$ is finite and that $I(f(X); g(Y)) = I(X; Y)$?

I know that for discrete random variables the situation is simpler and I tried to ask this question at MathSE, but it has not been answered there.

Any intuitions, counterexamples, or literature suggestions would be very much welcome.

Answer 1

It seems that the theorem is indeed true in the smooth case and follows from a more general result:

Let $\mathcal X$, $\mathcal X'$, $\mathcal Y$ and $\mathcal Y'$ be standard Borel spaces (this includes topological manifolds with their Borel $\sigma$-algebras) and $f\colon \mathcal X\to \mathcal X'$ and $g\colon \mathcal Y\to\mathcal Y'$ be continuous injective mappings.

Then, for every two random variables $X\colon \Omega\to \mathcal X$ and $Y\colon \Omega\to \mathcal Y$ $$I(X; Y) = I( f(X); g(Y)).$$

Proof: First, note that a continuous mapping between Borel spaces is measurable, so random variables $f(X)$ and $g(Y)$ are well-defined.

Then, observe that without loss of generality that it suffices to prove that $$I(f(X); Y) = I(X; Y).$$

A variant of data processing inequality asserts that $I(f(X); Y)\le I(X; Y)$ for every measurable mapping $f$, so we will need to prove the reverse inequality.

Recall that $$ I(X;Y) = \sup_{\mathcal E, \mathcal F} \sum_{i, j} P_{XY}(E_i\times F_j) \log \frac{P_{XY}(E_i\times F_j) }{ P_X(E_i) P_Y(F_j) }, $$ where the supremum is taken over all finite partitions $\mathcal E$ of $\mathcal X$ and $\mathcal F$ of $\mathcal Y$ and we again take the same conventions for dealing with zero probabilities as in the question.

The proof will go as follows: for every finite partition $\mathcal E$ of $\mathcal X$ we will construct a finite partition of $\mathcal X'$, which (together with an arbitrary partition $\mathcal F$ of $\mathcal Y$) will give the same value of the sum.

Take any finite partitions $\mathcal E$ of $\mathcal X$ and $\mathcal F$ of $\mathcal Y$. First, we will show that sets \begin{align*} G_0 &= \mathcal X'\setminus f(\mathcal X)\\ G_1 &= f(E_1)\\ G_2 &= f(E_2)\\ &~~\vdots \end{align*} form a partition of $\mathcal X'$.

It is easy to see that they are pairwise disjoint as $f$ is injective and it is obvious that their union is the whole $\mathcal X'$. Hence, we only need to show that they are all Borel.

For $i\ge 1$ it is clear that $G_i$ is Borel from Lusin–Suslin theorem (Theorem 15.1 of Kechris' Classical Descriptive Set Theory). Similarly, $f(\mathcal X)$ is Borel and hence $G_0 = \mathcal X'\setminus f(\mathcal X)$ is Borel as well.

Next, we will prove that

$$\sum_{i = 1}^n \sum_{j=1}^m P_{XY}(E_i\times F_j) \log \frac{P_{XY}(E_i\times F_j) }{ P_X(E_i) P_Y(F_j) } = \sum_{i = 0}^n \sum_{j=1}^m P_{f(X)Y}(G_i\times F_j) \log \frac{P_{f(X)Y}(G_i\times F_j) }{ P_{f(X)}(G_i) P_Y(F_j)}. $$

Note that $P_{f(X)}(G_0) = P_X(f^{-1}(G_0)) = P_X(\varnothing) = 0$, so that we can ignore the terms with $i=0$.

For $i\ge 1$ we have

$$P_{f(X)}(G_i) = P_X( f^{-1}(G_i) ) = P_X( f^{-1}(f( E_i)) ) = P_X(E_i)$$ as $f$ is injective.

Similarly, $f\times 1_{\mathcal Y}$ is injective and maps $E_i\times F_j$ onto $G_i\times F_j$, so $$P_{f(X)Y}( G_i\times F_j ) = P_{XY}( (f\times 1_{\mathcal Y})^{-1}( G_i\times F_j ) ) = P_{XY}(E_i\times F_j).$$

This finishes the proof.