Let $X\colon \Omega\to\mathcal X$ and $Y\colon \Omega\to \mathcal Y$ be two random variables. In M.S. Pinkser's Information and information stability of random variables and processes mutual information is defined as:

$$I(X; Y) = \sup \sum_i P_{(X,Y)}(E_i) \log \frac{ P_{(X, Y)}(E_i)}{ P_{X\times Y}(E_i),} ,$$ where the supremum is taken over all finite partitions $\{E_1, \dotsc, E_n\}$ of $\mathcal X\times\mathcal Y$ and one uses the usual conventions $0\cdot \log 0/a = a\log a/0 = 0$ for $a\ge 0$.

In particular, if $P_{(X, Y)}$ is not absolutely continuous w.r.t. $P_{X\times Y} = P_X\otimes P_Y$, then $I(X; Y) = \infty$. In the other case, mutual information becomes the well-known formula involving the Radon–Nikodym derivative:

$$I(X; Y) = \int_{\mathcal X\times \mathcal Y} \log \left( \frac{ \mathrm{d}P_{X,Y} }{ \mathrm{d}(P_X\otimes P_Y) } \right)\, \mathrm{d}P_{X,Y}$$ and is finite.

One can prove that if $f\colon \mathcal X\to \mathcal X'$ and $g\colon \mathcal Y\to \mathcal Y'$ are measurable bijective mapping with measurable inverses, then $I(X; Y) = I(f(X); g(Y))$.

I wonder if this assumption can be weakened in the smooth case:

Assume that $\mathcal X = \mathbb R^m$ and $\mathcal Y = \mathbb R^n$ and $I(X; Y)$ is finite. Let $f\colon \mathbb R^m\to \mathbb R^{m+k}$ and $g\colon \mathbb R^n\to \mathbb R^{n+l}$ be smooth embeddings.

Is this true that $I(f(X); g(Y))$ is finite and that $I(f(X); g(Y)) = I(X; Y)$?

I know that for discrete random variables the situation is simpler and I tried to ask this question at MathSE, but it has not been answered there.

Any intuitions, counterexamples, or literature suggestions would be very much welcome.


1 Answer 1


It seems that the theorem is indeed true in the smooth case and follows from a more general result:

Let $\mathcal X$, $\mathcal X'$, $\mathcal Y$ and $\mathcal Y'$ be standard Borel spaces (this includes topological manifolds with their Borel $\sigma$-algebras) and $f\colon \mathcal X\to \mathcal X'$ and $g\colon \mathcal Y\to\mathcal Y'$ be continuous injective mappings.

Then, for every two random variables $X\colon \Omega\to \mathcal X$ and $Y\colon \Omega\to \mathcal Y$ $$I(X; Y) = I( f(X); g(Y)).$$

Proof: First, note that a continuous mapping between Borel spaces is measurable, so random variables $f(X)$ and $g(Y)$ are well-defined.

Then, observe that without loss of generality that it suffices to prove that $$I(f(X); Y) = I(X; Y).$$

A variant of data processing inequality asserts that $I(f(X); Y)\le I(X; Y)$ for every measurable mapping $f$, so we will need to prove the reverse inequality.

Recall that $$ I(X;Y) = \sup_{\mathcal E, \mathcal F} \sum_{i, j} P_{XY}(E_i\times F_j) \log \frac{P_{XY}(E_i\times F_j) }{ P_X(E_i) P_Y(F_j) }, $$ where the supremum is taken over all finite partitions $\mathcal E$ of $\mathcal X$ and $\mathcal F$ of $\mathcal Y$ and we again take the same conventions for dealing with zero probabilities as in the question.

The proof will go as follows: for every finite partition $\mathcal E$ of $\mathcal X$ we will construct a finite partition of $\mathcal X'$, which (together with an arbitrary partition $\mathcal F$ of $\mathcal Y$) will give the same value of the sum.

Take any finite partitions $\mathcal E$ of $\mathcal X$ and $\mathcal F$ of $\mathcal Y$. First, we will show that sets \begin{align*} G_0 &= \mathcal X'\setminus f(\mathcal X)\\ G_1 &= f(E_1)\\ G_2 &= f(E_2)\\ &~~\vdots \end{align*} form a partition of $\mathcal X'$.

It is easy to see that they are pairwise disjoint as $f$ is injective and it is obvious that their union is the whole $\mathcal X'$. Hence, we only need to show that they are all Borel.

For $i\ge 1$ it is clear that $G_i$ is Borel from Lusin–Suslin theorem (Theorem 15.1 of Kechris' Classical Descriptive Set Theory). Similarly, $f(\mathcal X)$ is Borel and hence $G_0 = \mathcal X'\setminus f(\mathcal X)$ is Borel as well.

Next, we will prove that

$$\sum_{i = 1}^n \sum_{j=1}^m P_{XY}(E_i\times F_j) \log \frac{P_{XY}(E_i\times F_j) }{ P_X(E_i) P_Y(F_j) } = \sum_{i = 0}^n \sum_{j=1}^m P_{f(X)Y}(G_i\times F_j) \log \frac{P_{f(X)Y}(G_i\times F_j) }{ P_{f(X)}(G_i) P_Y(F_j)}. $$

Note that $P_{f(X)}(G_0) = P_X(f^{-1}(G_0)) = P_X(\varnothing) = 0$, so that we can ignore the terms with $i=0$.

For $i\ge 1$ we have

$$P_{f(X)}(G_i) = P_X( f^{-1}(G_i) ) = P_X( f^{-1}(f( E_i)) ) = P_X(E_i)$$ as $f$ is injective.

Similarly, $f\times 1_{\mathcal Y}$ is injective and maps $E_i\times F_j$ onto $G_i\times F_j$, so $$P_{f(X)Y}( G_i\times F_j ) = P_{XY}( (f\times 1_{\mathcal Y})^{-1}( G_i\times F_j ) ) = P_{XY}(E_i\times F_j).$$

This finishes the proof.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.