Let $A$, $B$ and $C$ be three random variables and $p_{A,B,C}=p_Ap_Bp_{C|A,B}$ and $q_{A,B,C}=p_Aq_{B|A}p_{C|A,B}$ be two distributions on them. Then, we can conclude that? \begin{align*} \lvert I_p(A,B;C)-I_q(A,B;C)\rvert\leq\log\lvert\mathcal{B}\rvert. \end{align*} where $\mathcal{B}$ is the alphabet set of $B$ and $I(.;.)$ is the mutual information.

  • $\begingroup$ By $AB$, do you mean the vector $(A,B)$ or do you mean the product? And there seems to be a typo in your definition of $q$. $\endgroup$ – Pat Devlin Jan 27 '17 at 0:56
  • $\begingroup$ I mean $(A,B)$. No, only instead of $p_B$ we have $q_{B|A}$. $\endgroup$ – Math_Y Jan 27 '17 at 6:30
  • $\begingroup$ What are $p_A$ and $q_{B|A}$? $\endgroup$ – Pat Devlin Jan 27 '17 at 12:26
  • $\begingroup$ $p_A$ is pmf of random variable $A$ and $q_{B|A}$ is conditional pmf of $B$ conditioned on $A$. $\endgroup$ – Math_Y Jan 27 '17 at 14:29
  • $\begingroup$ How does that differ from $p_{B|A}?$ $\endgroup$ – Pat Devlin Jan 27 '17 at 14:36

The set-up is the following.

We have random variables $A,B,C$ and $A', B', C'$. We know $A$ and $A'$ have the same distribution, and for all $(a,b)$, the random variable $C | (A,B)=(a,b)$ has the same distribution as $C' | (A', B') = (a,b)$.

Under the above hypothesis, the claim you want is in fact false. Here's a counterexample.

Let $A$ and $A'$ be uniformly distributed over $\{1, 2, \ldots, a\}$. Let $B$ and $B'$ be independently drawn from $\{0,1\}$ such that $\mathbb{P}(B=0) = \mathbb{P}(B'=1) = \varepsilon$. And let $C = AB$ and $C' = A' B'$ be the product of the other two.

(I'm allowing the $\varepsilon$ in there just so that we don't have to condition on any events having probability $0$ in the definitions of $C$ and $C'$, but you should imagine $\varepsilon \approx 0$. And every $\approx$ below becomes equality in the limit as $\varepsilon \to 0$.)

Then we have $H(A,B) = H(A', B') = H(A) + H(B) \approx H(A) = \log(a)$. On the other hand, $H(A,B | C) \approx 0$ since with probability $1-\varepsilon$, $(A,B)$ is uniquely determined by $C$. Similarly $H(A', B' | C') \approx H(A')$ since $C'$ usually tells us nothing about $A'$. This give us

$$ I(A, B; C) - I(A', B'; C') = H(A,B) - H(A,B|C) - [H(A',B') - H(A',B'|C')] = H(A',B'|C') - H(A,B|C) \approx \log(a), $$ which can be made arbitrarily large.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.