## Problem Definition ##
Let $\mathbf{G}$ and $\mathbf{S}$ be jointly distributed random variables where
$\mathbf{S}$ is continuous and is related to $\mathbf{G}$ through a conditional pdf $f(s|g)$ defined for all g. 
The conditional differential entropy of $\mathbf{S}$ given $\mathbf{G}$ is defined as
\begin{align}\label{hsg}
h(\mathbf{S}|\mathbf{G}) &= -\int_{\mathcal{G,S}}f(s,g)\log(f(s|g))dsdg \nonumber \\
&= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg.
\end{align}
If $f(\mathbf{S}|\mathbf{G}=g)$ is a Gaussian distribution centred at $g$, we want to proof the following expression is true:
\begin{align}
\label{definition_s_g}
\color{blue}
{
    h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0
}
\end{align}

This is our idea to prove it, could someone check if our proof is correct?

----------


## Our Proof ##

Using the above conditional entropy definition, $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ can be rewritten as:
\begin{align}
h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg \nonumber \\ &- \int_{\mathcal{G,S}}f(s,g)\log(\frac{p(s)}{f(s,g)})dsdg \nonumber \\
&= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)} \frac{f(s,g)}{f(s)})dsdg \nonumber\\
&= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s)})dsdg.
\end{align}

If we want to prove $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ is positive, we can prove that $p(g)/p(s)$ is greater than $1$. Since $f(s,g)$ is always positive, if we can prove that $log(\frac{f(g)}{f(s)})$ is also positive for any given $s$ and $g$, we can say the integral is also positive. 

The likelihood of any $x\in \mathcal{S}$ can be computed by
\begin{equation}
f(x)=\int_{\mathcal{G}}f(\mathbf{S}=x|\mathbf{G}=g)f(\mathbf{G}=g)dg.
\end{equation}
For any $y\in \mathcal{G}$, we can write $p(y)$ in the same way as
\begin{equation}
f(y)=\int_{\mathcal{G}}f(\mathbf{G}=y|\mathbf{G}=g)f(\mathbf{G}=g)dg.
\end{equation}
Then, $f(y)/f(x)$ equals to
\begin{equation}
\frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg.
\label{equ:py_ps}
\end{equation}

Since $f(y|g)=0$ when $g\neq y$, we can further simplify the equation as: 
\begin{equation}
\frac{f(y)}{f(x)} = \frac{f(y|g=y)}{f(x|g=y)} = \frac{\delta(y)}{f(x|g=y)} = \frac{+\infty}{f(x|g=y)},
\end{equation}
where $\delta(\cdot)$ is a Dirac delta function.

As $\mathbf{S}$ is Gaussian distributed around $g$, given $g=y$, for any given $x \in \mathbf{S}$ and $y \in \mathbf{G}$, $f(x|g=y)$ is always smaller than $+\infty$, then $\frac{p(y)}{p(x)}$ is always bigger than $1$. This means $\log(\frac{p(y)}{p(x)})$ is positive and  $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$ is positive as well. As a result, 
$\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$


----------
## Update 1 ##
Now I see this proof is wrong because integral does not distribute over fraction. So we cannot get
\begin{equation}
\frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg.
\end{equation}

Are there other solutions?

----------

----------
## Update 2: Modification on the problem definition ##
In order to simplify the problem we give additional constrains on \mathbf{G}:

 - $\mathbf{G}$ takes N possible values and follows **a discrete uniform distribution**

The rests are the same, $\mathbf{S}$ is continuous and is related to $\mathbf{G}$ through a conditional pdf $f(\mathbf{S}|\mathbf{G}=g)=\mathcal{N}(\mu_g, \sigma^2)$ centred at different $g$.

The proves are:
\begin{align}
h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) =\sum_{\mathcal{G}} \int_{\mathcal{S}}f(s,g)\log(\frac{f_M(g)}{f(s)})ds
\end{align}
where $f_M(g) = \frac{1}{N}$ is the probability mass function for taking each $g$.

Using the law of total probability, for any $x$ in $\mathcal{S}$ and $y$ in $\mathcal{G}$:
\begin{equation}
f(x)=\sum_{\mathcal{G}}f(x|g)f_M(g).
\end{equation}
\begin{equation}
f(y)=\sum_{\mathcal{G}}f(y|g)f_M(g)=\sum_{\mathcal{G}}\delta(y)f_M(g).
\end{equation}

Then, since $f_M(g)$ is constant and can be cancelled, $f(y)/f(x)$ equals to
\begin{equation}
\frac{f(y)}{f(x)} =  \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)}.
\end{equation}

Form the property of Gaussian distribution we know that $f(x|g)$ is maximum when $x=g$, we estimate the lover bound of the above equation as:
\begin{align}
\frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)} &\geq \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(\mu_g|g)}  
= \frac{\delta(y)}{Nf(\mu_g|g)}  
= \frac{+\infty}{Nf(\mu_g|g)} \nonumber \\
&= \frac{+\infty}{\frac{N}{\sigma\sqrt{2\pi}}}
\end{align}

As $\mathbf{S}$ is Gaussian distributed around $g$, given $g$ is at the same location as $y$, for any given $x$ and $y$, $f(\mu_g|g)$ is always smaller than $+\infty$ and $\frac{f(y)}{f(x)}$ is always bigger than $1$. This means $\log(\frac{f(y)}{f(x)})$ is positive and  $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$ is positive as well. As a result, 
$\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$

**Is the proof correct this time?**