Problem Definition
Let $\mathbf{G}$ and $\mathbf{S}$ be jointly distributed random variables where $\mathbf{S}$ is continuous and is related to $\mathbf{G}$ through a conditional pdf $f(s|g)$ defined for all g. The conditional differential entropy of $\mathbf{S}$ given $\mathbf{G}$ is defined as \begin{align}\label{hsg} h(\mathbf{S}|\mathbf{G}) &= -\int_{\mathcal{G,S}}f(s,g)\log(f(s|g))dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg. \end{align} If $f(\mathbf{S}|\mathbf{G}=g)$ is a Gaussian distribution centred at $g$, we want to proof the following expression is true: \begin{align} \label{definition_s_g} \color{blue} { h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0 } \end{align}
This is our idea to prove it, could someone check if our proof is correct?
Our Proof
Using the above conditional entropy definition, $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ can be rewritten as: \begin{align} h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg \nonumber \\ &- \int_{\mathcal{G,S}}f(s,g)\log(\frac{p(s)}{f(s,g)})dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)} \frac{f(s,g)}{f(s)})dsdg \nonumber\\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s)})dsdg. \end{align}
If we want to prove $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ is positive, we can prove that $p(g)/p(s)$ is greater than $1$. Since $f(s,g)$ is always positive, if we can prove that $log(\frac{f(g)}{f(s)})$ is also positive for any given $s$ and $g$, we can say the integral is also positive.
The likelihood of any $x\in \mathcal{S}$ can be computed by \begin{equation} f(x)=\int_{\mathcal{G}}f(\mathbf{S}=x|\mathbf{G}=g)f(\mathbf{G}=g)dg. \end{equation} For any $y\in \mathcal{G}$, we can write $p(y)$ in the same way as \begin{equation} f(y)=\int_{\mathcal{G}}f(\mathbf{G}=y|\mathbf{G}=g)f(\mathbf{G}=g)dg. \end{equation} Then, $f(y)/f(x)$ equals to \begin{equation} \frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg. \label{equ:py_ps} \end{equation}
Since $f(y|g)=0$ when $g\neq y$, we can further simplify the equation as: \begin{equation} \frac{f(y)}{f(x)} = \frac{f(y|g=y)}{f(x|g=y)} = \frac{\delta(y)}{f(x|g=y)} = \frac{+\infty}{f(x|g=y)}, \end{equation} where $\delta(\cdot)$ is a Dirac delta function.
As $\mathbf{S}$ is Gaussian distributed around $g$, given $g=y$, for any given $x \in \mathbf{S}$ and $y \in \mathbf{G}$, $f(x|g=y)$ is always smaller than $+\infty$, then $\frac{p(y)}{p(x)}$ is always bigger than $1$. This means $\log(\frac{p(y)}{p(x)})$ is positive and $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$ is positive as well. As a result, $\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$
Update 1
Now I see this proof is wrong because integral does not distribute over fraction. So we cannot get \begin{equation} \frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg. \end{equation}
Are there other solutions?
Update 2: Modification on the problem definition
In order to simplify the problem we give additional constrains on \mathbf{G}:
- $\mathbf{G}$ takes N possible values and follows a discrete uniform distribution
The rests are the same, $\mathbf{S}$ is continuous and is related to $\mathbf{G}$ through a conditional pdf $f(\mathbf{S}|\mathbf{G}=g)=\mathcal{N}(\mu_g, \sigma^2)$ centred at different $g$.
The proves are: \begin{align} h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) =\sum_{\mathcal{G}} \int_{\mathcal{S}}f(s,g)\log(\frac{f_M(g)}{f(s)})ds \end{align} where $f_M(g) = \frac{1}{N}$ is the probability mass function for taking each $g$.
Using the law of total probability, for any $x$ in $\mathcal{S}$ and $y$ in $\mathcal{G}$: \begin{equation} f(x)=\sum_{\mathcal{G}}f(x|g)f_M(g). \end{equation} \begin{equation} f(y)=\sum_{\mathcal{G}}f(y|g)f_M(g)=\sum_{\mathcal{G}}\delta(y)f_M(g). \end{equation}
Then, since $f_M(g)$ is constant and can be cancelled, $f(y)/f(x)$ equals to \begin{equation} \frac{f(y)}{f(x)} = \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)}. \end{equation}
Form the property of Gaussian distribution we know that $f(x|g)$ is maximum when $x=g$, we estimate the lover bound of the above equation as: \begin{align} \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)} &\geq \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(\mu_g|g)} = \frac{\delta(y)}{Nf(\mu_g|g)} = \frac{+\infty}{Nf(\mu_g|g)} \nonumber \\ &= \frac{+\infty}{\frac{N}{\sigma\sqrt{2\pi}}} \end{align}
As $\mathbf{S}$ is Gaussian distributed around $g$, given $g$ is at the same location as $y$, for any given $x$ and $y$, $f(\mu_g|g)$ is always smaller than $+\infty$ and $\frac{f(y)}{f(x)}$ is always bigger than $1$. This means $\log(\frac{f(y)}{f(x)})$ is positive and $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$ is positive as well. As a result, $\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$
Is the proof correct this time?
