Skip to main content
1 of 2
math-Student
  • 1.1k
  • 7
  • 12

An inequality involving conditional variance and its connection to information theory

Given absolutely continuous random variables $(X, Y)$ with joint distribution $P_{XY}$, we construct $Z:=\sqrt{\gamma} Y+N_\mathsf{G}$ where $N_\mathsf{G}\sim N(0, 1)$ and is independent of $(X,Y)$ and $\gamma\geq 0$. The conditional variance of $Y$ given $Z$ is defined as $$\mathsf{var}(Y|Z)=\mathbb{E}[(Y-\mathbb{E}[Y|Z])^2|Z],$$ and hence $$\mathbb{E}[\mathsf{var}(Y|Z)]=\mathbb{E}[(Y-\mathbb{E}[Y|Z])^2].$$ What are known about conditional variance:

  1. Independence: If $Y$ is independent of Z (i.e., $\gamma=0$), then $\mathbb{E}[\mathsf{var}(Y|Z)]=\mathsf{var}(Y)$ and if $Y=Z$, then $\mathbb{E}[\mathsf{var}(Y|Z)]=0$. In general, $0\leq \mathbb{E}[\mathsf{var}(Y|Z)]\leq \mathsf{var}(Y)$.
  2. Law of total variance: $\mathsf{var}(Y)=\mathbb{E}[\mathsf{var}(Y|Z)]+\mathsf{var}(\mathbb{E}[Y|Z])$.
  3. Orthogonality: $\mathbb{E}[\mathsf{var}(Y|Z, X)]\leq \mathbb{E}[\mathsf{var}(Y|Z)]$.
  4. Data Processing inequality: Since $(X, Y, Z)$ form the Markov chain $X\leftrightarrow Y\leftrightarrow Z$, we have $\mathbb{E}[\mathsf{var}(X|Y)]\leq \mathbb{E}[\mathsf{var}(X|Z)]$.

In engineering, $\mathbb{E}[\mathsf{var}(Y|Z)]$ is called minimum mean-squared error of estimating $Y$ (the signal) given $Z$ (the noisy observation). Hence, when $\gamma\to \infty$, then the noise contaminating the signal is negligible which mean $Z$ can be used to perfectly estimate $Y$ and mathematically $\mathbb{E}[\mathsf{var}(Y|Z)]=0$.

We want to show that $$\frac{\mathbb{E}[\mathsf{var}(Y|Z, X)]}{\mathbb{E}[\mathsf{var}(Y|Z)]}\geq \frac{\mathbb{E}[\mathsf{var}(Y|X)]}{\mathsf{var}(Y)}.$$

Why is this inequality important?

Once proved, it implies that $$I(X; Z)\leq \left(1-\frac{\mathbb{E}[\mathsf{var}(Y|X)]}{\mathsf{var}(Y)}\right)I(Y; Z),$$ where $I(X; Z)$ represents the mutual information between random variable $X$ and $Z$. This inequality is a very strong version of the well-known "data processing inequality" for mutual information (which states $I(X; Z)\leq I(Y;Z)$ for any $\gamma\geq 0$).

math-Student
  • 1.1k
  • 7
  • 12