# Bounding difference of two mutual information with different distributions on random variables

Let $A$, $B$ and $C$ be three random variables and $p_{A,B,C}=p_Ap_Bp_{C|A,B}$ and $q_{A,B,C}=p_Aq_{B|A}p_{C|A,B}$ be two distributions on them. Then, we can conclude that? \begin{align*} \lvert I_p(A,B;C)-I_q(A,B;C)\rvert\leq\log\lvert\mathcal{B}\rvert. \end{align*} where $\mathcal{B}$ is the alphabet set of $B$ and $I(.;.)$ is the mutual information.

• By $AB$, do you mean the vector $(A,B)$ or do you mean the product? And there seems to be a typo in your definition of $q$. – Pat Devlin Jan 27 '17 at 0:56
• I mean $(A,B)$. No, only instead of $p_B$ we have $q_{B|A}$. – Math_Y Jan 27 '17 at 6:30
• What are $p_A$ and $q_{B|A}$? – Pat Devlin Jan 27 '17 at 12:26
• $p_A$ is pmf of random variable $A$ and $q_{B|A}$ is conditional pmf of $B$ conditioned on $A$. – Math_Y Jan 27 '17 at 14:29
• How does that differ from $p_{B|A}?$ – Pat Devlin Jan 27 '17 at 14:36

The set-up is the following.

We have random variables $A,B,C$ and $A', B', C'$. We know $A$ and $A'$ have the same distribution, and for all $(a,b)$, the random variable $C | (A,B)=(a,b)$ has the same distribution as $C' | (A', B') = (a,b)$.

Under the above hypothesis, the claim you want is in fact false. Here's a counterexample.

Let $A$ and $A'$ be uniformly distributed over $\{1, 2, \ldots, a\}$. Let $B$ and $B'$ be independently drawn from $\{0,1\}$ such that $\mathbb{P}(B=0) = \mathbb{P}(B'=1) = \varepsilon$. And let $C = AB$ and $C' = A' B'$ be the product of the other two.

(I'm allowing the $\varepsilon$ in there just so that we don't have to condition on any events having probability $0$ in the definitions of $C$ and $C'$, but you should imagine $\varepsilon \approx 0$. And every $\approx$ below becomes equality in the limit as $\varepsilon \to 0$.)

Then we have $H(A,B) = H(A', B') = H(A) + H(B) \approx H(A) = \log(a)$. On the other hand, $H(A,B | C) \approx 0$ since with probability $1-\varepsilon$, $(A,B)$ is uniquely determined by $C$. Similarly $H(A', B' | C') \approx H(A')$ since $C'$ usually tells us nothing about $A'$. This give us

$$I(A, B; C) - I(A', B'; C') = H(A,B) - H(A,B|C) - [H(A',B') - H(A',B'|C')] = H(A',B'|C') - H(A,B|C) \approx \log(a),$$ which can be made arbitrarily large.