Given absolutely continuous random variables $(X, Y)$ with joint distribution $P_{XY}$, we construct $Z:=\sqrt{\gamma} Y+N_\mathsf{G}$ where $N_\mathsf{G}\sim N(0, 1)$ and is independent of $(X,Y)$ and $\gamma\geq 0$. The conditional variance of $Y$ given $Z$ is defined as $$\mathsf{var}(Y|Z)=\mathbb{E}[(Y-\mathbb{E}[Y|Z])^2|Z],$$ and hence $$\mathbb{E}[\mathsf{var}(Y|Z)]=\mathbb{E}[(Y-\mathbb{E}[Y|Z])^2].$$ What are known about conditional variance:

  1. Independence: If $Y$ is independent of Z (i.e., $\gamma=0$), then $\mathbb{E}[\mathsf{var}(Y|Z)]=\mathsf{var}(Y)$ and if $Y=Z$, then $\mathbb{E}[\mathsf{var}(Y|Z)]=0$. In general, $0\leq \mathbb{E}[\mathsf{var}(Y|Z)]\leq \mathsf{var}(Y)$.
  2. Law of total variance: $\mathsf{var}(Y)=\mathbb{E}[\mathsf{var}(Y|Z)]+\mathsf{var}(\mathbb{E}[Y|Z])$.
  3. Orthogonality: $\mathbb{E}[\mathsf{var}(Y|Z, X)]\leq \mathbb{E}[\mathsf{var}(Y|Z)]$.
  4. Data Processing inequality: Since $(X, Y, Z)$ form the Markov chain $X\leftrightarrow Y\leftrightarrow Z$, we have $\mathbb{E}[\mathsf{var}(X|Y)]\leq \mathbb{E}[\mathsf{var}(X|Z)]$.

In engineering, $\mathbb{E}[\mathsf{var}(Y|Z)]$ is called minimum mean-squared error of estimating $Y$ (the signal) given $Z$ (the noisy observation). Hence, when $\gamma\to \infty$, then the noise contaminating the signal is negligible which mean $Z$ can be used to perfectly estimate $Y$ and mathematically $\mathbb{E}[\mathsf{var}(Y|Z)]=0$.

We want to show that for any $\gamma\geq 0$ $$\frac{\mathbb{E}[\mathsf{var}(Y|Z, X)]}{\mathbb{E}[\mathsf{var}(Y|Z)]}\geq \frac{\mathbb{E}[\mathsf{var}(Y|X)]}{\mathsf{var}(Y)}.$$

It is not hard to show that for jointly Gaussian $(X,Y)$ this inequality holds for any $\gamma\geq 0$.

Why is this inequality important?

Once proved, it implies that $$I(X; Z)\leq \left(1-\frac{\mathbb{E}[\mathsf{var}(Y|X)]}{\mathsf{var}(Y)}\right)I(Y; Z),$$ where $I(X; Z)$ represents the mutual information between random variable $X$ and $Z$. This inequality is a very strong version of the well-known "data processing inequality" for mutual information (which states $I(X; Z)\leq I(Y;Z)$ for any $\gamma\geq 0$).

  • $\begingroup$ Is the strong data processing inequality you mention open? $\endgroup$ – kodlu Mar 31 '16 at 2:18
  • $\begingroup$ Yes, this version of strong data processing inequality is still open. $\endgroup$ – math-Student Apr 7 '16 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.