If the mutual information $I(X;Y)$ is high, how can we prove that if $I(X;Z)$ is high then $I(Y;Z)$ is high too? Let say, we have three random variables, $X$, $Y$, and $Z$, where $X$ and $Y$ have high mutual information $I(X;Y)$. Then, how can we prove that if $I(X;Z)$ is high then $I(Y;Z)$ is high too??
Any suggestions, references or direct solutions are welcome!
 A: I don't think any such statement can be true in general.
For a real number $x \in [0,1]$ with binary expansion $0.x_1 x_2 x_3 x_4 \dots$, let $f(x) = 0.x_1 x_3 x_5\dots$ and $g(x) = 0.x_2 x_4 x_6 \dots$.  Let $X \sim U(0,1)$ and let $Y = f(X)$, $Z = g(X)$.  Note that $Y,Z \sim U(0,1)$ as well.  Now the joint distribution of $(X,Y)$ is supported on the graph of $f$, which is a measure-zero subset of $[0,1]^2$, so it is singular with respect to the product measure $U(0,1) \otimes U(0,1)$.  Therefore $I(X;Y) = +\infty$, and likewise $I(X;Z)= +\infty$ as well.  But $Y,Z$ are independent so $I(Y;Z) = 0$.
If you want to stick with mutually absolutely continuous measures, you should be able to "fuzz" this example: let $\xi, \eta \sim N(0,1)$ be independent of $X$ and each other, and take $Y = f(X) + \epsilon \xi$, $Z = g(X) + \epsilon \eta$.  By letting $\epsilon$ become small, you should be able to get $I(X;Y)$ and $I(X;Z)$ as large as you want (while still finite), and still $I(Y;Z)=0$.
A: This answer is fundamentally similar to Nate's but looks different.
Fix a finite set $E$ and subsets $A, B, C$.  Choose a subset $S$ of $E$ uniformly at random, and put
$$
X = S \cap A, \quad Y = S \cap B, \quad Z = S \cap C.
$$
Then
$$
I(X; Y) = |A \cap B|, \quad I(X; Z) = |A \cap C|, \quad I(Y; Z) = |B \cap C|.
$$
So in this case, you're asking:

If $|A \cap B|$ and $|A \cap C|$ are large, is $|B \cap C|$?

And the answer is no, e.g. it's easy to think of examples where
$$
|A \cap B| = |A \cap C| = 10^{10^{10}}, \quad |B \cap C| = 0.
$$
But as user44191 comments, if by "large" you mean large relative to $H(X)$, $H(Y)$ and $H(Z)$ (which here are $|A|$, $|B|$ and $|C|$) then it's a different matter.  Maybe you can clarify.
