Let $A$, $B$ and $C$ be three random variables and $p_{A,B,C}=p_Ap_Bp_{C|A,B}$ and $q_{A,B,C}=p_Aq_{B|A}p_{C|A,B}$ be two distributions on them. Then, we can conclude that? \begin{align*} \lvert I_p(A,B;C)-I_q(A,B;C)\rvert\leq\log\lvert\mathcal{B}\rvert. \end{align*} where $\mathcal{B}$ is the alphabet set of $B$ and $I(.;.)$ is the mutual information.
$\begingroup$
$\endgroup$
8
-
$\begingroup$ By $AB$, do you mean the vector $(A,B)$ or do you mean the product? And there seems to be a typo in your definition of $q$. $\endgroup$– Pat DevlinCommented Jan 27, 2017 at 0:56
-
$\begingroup$ I mean $(A,B)$. No, only instead of $p_B$ we have $q_{B|A}$. $\endgroup$– Math_YCommented Jan 27, 2017 at 6:30
-
$\begingroup$ What are $p_A$ and $q_{B|A}$? $\endgroup$– Pat DevlinCommented Jan 27, 2017 at 12:26
-
$\begingroup$ $p_A$ is pmf of random variable $A$ and $q_{B|A}$ is conditional pmf of $B$ conditioned on $A$. $\endgroup$– Math_YCommented Jan 27, 2017 at 14:29
-
$\begingroup$ How does that differ from $p_{B|A}?$ $\endgroup$– Pat DevlinCommented Jan 27, 2017 at 14:36
|
Show 3 more comments
1 Answer
$\begingroup$
$\endgroup$
The set-up is the following.
We have random variables $A,B,C$ and $A', B', C'$. We know $A$ and $A'$ have the same distribution, and for all $(a,b)$, the random variable $C | (A,B)=(a,b)$ has the same distribution as $C' | (A', B') = (a,b)$.
Under the above hypothesis, the claim you want is in fact false. Here's a counterexample.
Let $A$ and $A'$ be uniformly distributed over $\{1, 2, \ldots, a\}$. Let $B$ and $B'$ be independently drawn from $\{0,1\}$ such that $\mathbb{P}(B=0) = \mathbb{P}(B'=1) = \varepsilon$. And let $C = AB$ and $C' = A' B'$ be the product of the other two.
(I'm allowing the $\varepsilon$ in there just so that we don't have to condition on any events having probability $0$ in the definitions of $C$ and $C'$, but you should imagine $\varepsilon \approx 0$. And every $\approx$ below becomes equality in the limit as $\varepsilon \to 0$.)
Then we have $H(A,B) = H(A', B') = H(A) + H(B) \approx H(A) = \log(a)$. On the other hand, $H(A,B | C) \approx 0$ since with probability $1-\varepsilon$, $(A,B)$ is uniquely determined by $C$. Similarly $H(A', B' | C') \approx H(A')$ since $C'$ usually tells us nothing about $A'$. This give us
$$ I(A, B; C) - I(A', B'; C') = H(A,B) - H(A,B|C) - [H(A',B') - H(A',B'|C')] = H(A',B'|C') - H(A,B|C) \approx \log(a), $$ which can be made arbitrarily large.
