Pblblem Definition

Let $\mathbf{G}$ and $\mathbf{S}$ be jointly distributed random variables where $\mathbf{S}$ is continuous and is related to $\mathbf{G}$ through a conditional pdf $f(s|g)$ defined for all g. The conditional differential entropy of $\mathbf{S}$ given $\mathbf{G}$ is defined as \begin{align}\label{hsg} h(\mathbf{S}|\mathbf{G}) &= -\int_{\mathcal{G,S}}f(s,g)\log(f(s|g))dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg. \end{align} If $f(\mathbf{S}|\mathbf{G}=g)$ is a Gaussian distribution centred at $g$, we want to proof the following expression is true: \begin{align} \label{definition_s_g} \color{blue} { h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0 } \end{align}

This is our idea to prove it, could someone check if our proof is correct?

Our Proof

Using the above conditional entropy definition, $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ can be rewritten as: \begin{align} h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg \nonumber \\ &- \int_{\mathcal{G,S}}f(s,g)\log(\frac{p(s)}{f(s,g)})dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)} \frac{f(s,g)}{f(s)})dsdg \nonumber\\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s)})dsdg. \end{align}

If we want to prove $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ is positive, we can prove that $p(g)/p(s)$ is greater than $1$. Since $f(s,g)$ is always positive, if we can prove that $log(\frac{f(g)}{f(s)})$ is also positive for any given $s$ and $g$, we can say the integral is also positive.

The likelihood of any $x\in \mathcal{S}$ can be computed by \begin{equation} f(x)=\int_{\mathcal{G}}f(\mathbf{S}=x|\mathbf{G}=g)f(\mathbf{G}=g)dg. \end{equation} For any $y\in \mathcal{G}$, we can write $p(y)$ in the same way as \begin{equation} f(y)=\int_{\mathcal{G}}f(\mathbf{G}=y|\mathbf{G}=g)f(\mathbf{G}=g)dg. \end{equation} Then, $f(y)/f(x)$ equals to \begin{equation} \frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg. \label{equ:py_ps} \end{equation}

Since $f(y|g)=0$ when $g\neq y$, we can further simplify the equation as: \begin{equation} \frac{f(y)}{f(x)} = \frac{f(y|g=y)}{f(x|g=y)} = \frac{\delta(y)}{f(x|g=y)} = \frac{+\infty}{f(x|g=y)}, \end{equation} where $\delta(\cdot)$ is a Dirac delta function.

As $\mathbf{S}$ is Gaussian distributed around $g$, given $g=y$, for any given $x \in \mathbf{S}$ and $y \in \mathbf{G}$, $f(x|g=y)$ is always smaller than $+\infty$, then $\frac{p(y)}{p(x)}$ is always bigger than $1$. This means $\log(\frac{p(y)}{p(x)})$ is positive and $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$ is positive as well. As a result, $\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$