Consider the discrete random variables $X,Y,Z,W$, where $Y$ and $W$ are independent. Let $Y = f(X)$, for some function $f(\cdot)$, $Z=Y+W$, and $X \rightarrow Y \rightarrow Z$ be a Markov chain.

By definition, the mutual information satisfies $I(Y;Z) = H(Y) - H(Y|Z) = H(Z) - H(Z|Y)$.

I know that because $Z=Y+W$ and $Y$ and $W$ are independent, then $H(Z|Y) = H(Y+W|Y) = H(W|Y) = H(W)$.

However, I found someone claiming in a paper that $H(Y|Z) = H(Z-W|Z) = H(W|Z) = H(W)$. I cannot understand this because $W$ and $Z$ are correlated. So why is it true that $H(W|Z) = H(W)$? Is it because of the Markov chain assumption?

Any insight would be greatly appreciated! Thanks for reading me.

  • 1
    $\begingroup$ You're right. If $Y$ takes the values 0 and 2 with equal probability and $W$ takes the values 0 and 1 with equal probability, independently of $Y$, then $H(Y|Z)$ is 0 since $Z$ determines $Y$. $\endgroup$ Mar 6, 2018 at 17:40
  • $\begingroup$ To the above, in your assumptions, $H(Y|Z)$ is $0$ but $H(W)=1$, wouldn't that contradict $H(Y|Z) = H(W)$? $\endgroup$
    – Lazy Lee
    Mar 14, 2018 at 2:33


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.