2
$\begingroup$

Problem Definition

Let $\mathbf{G}$ and $\mathbf{S}$ be jointly distributed random variables where $\mathbf{S}$ is continuous and is related to $\mathbf{G}$ through a conditional pdf $f(s|g)$ defined for all g. The conditional differential entropy of $\mathbf{S}$ given $\mathbf{G}$ is defined as \begin{align}\label{hsg} h(\mathbf{S}|\mathbf{G}) &= -\int_{\mathcal{G,S}}f(s,g)\log(f(s|g))dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg. \end{align} If $f(\mathbf{S}|\mathbf{G}=g)$ is a Gaussian distribution centred at $g$, we want to proof the following expression is true: \begin{align} \label{definition_s_g} \color{blue} { h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0 } \end{align}

This is our idea to prove it, could someone check if our proof is correct?


Our Proof

Using the above conditional entropy definition, $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ can be rewritten as: \begin{align} h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg \nonumber \\ &- \int_{\mathcal{G,S}}f(s,g)\log(\frac{p(s)}{f(s,g)})dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)} \frac{f(s,g)}{f(s)})dsdg \nonumber\\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s)})dsdg. \end{align}

If we want to prove $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$ is positive, we can prove that $p(g)/p(s)$ is greater than $1$. Since $f(s,g)$ is always positive, if we can prove that $log(\frac{f(g)}{f(s)})$ is also positive for any given $s$ and $g$, we can say the integral is also positive.

The likelihood of any $x\in \mathcal{S}$ can be computed by \begin{equation} f(x)=\int_{\mathcal{G}}f(\mathbf{S}=x|\mathbf{G}=g)f(\mathbf{G}=g)dg. \end{equation} For any $y\in \mathcal{G}$, we can write $p(y)$ in the same way as \begin{equation} f(y)=\int_{\mathcal{G}}f(\mathbf{G}=y|\mathbf{G}=g)f(\mathbf{G}=g)dg. \end{equation} Then, $f(y)/f(x)$ equals to \begin{equation} \frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg. \label{equ:py_ps} \end{equation}

Since $f(y|g)=0$ when $g\neq y$, we can further simplify the equation as: \begin{equation} \frac{f(y)}{f(x)} = \frac{f(y|g=y)}{f(x|g=y)} = \frac{\delta(y)}{f(x|g=y)} = \frac{+\infty}{f(x|g=y)}, \end{equation} where $\delta(\cdot)$ is a Dirac delta function.

As $\mathbf{S}$ is Gaussian distributed around $g$, given $g=y$, for any given $x \in \mathbf{S}$ and $y \in \mathbf{G}$, $f(x|g=y)$ is always smaller than $+\infty$, then $\frac{p(y)}{p(x)}$ is always bigger than $1$. This means $\log(\frac{p(y)}{p(x)})$ is positive and $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$ is positive as well. As a result, $\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$


Update 1

Now I see this proof is wrong because integral does not distribute over fraction. So we cannot get \begin{equation} \frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg. \end{equation}

Are there other solutions?



Update 2: Modification on the problem definition

In order to simplify the problem we give additional constrains on \mathbf{G}:

  • $\mathbf{G}$ takes N possible values and follows a discrete uniform distribution

The rests are the same, $\mathbf{S}$ is continuous and is related to $\mathbf{G}$ through a conditional pdf $f(\mathbf{S}|\mathbf{G}=g)=\mathcal{N}(\mu_g, \sigma^2)$ centred at different $g$.

The proves are: \begin{align} h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) =\sum_{\mathcal{G}} \int_{\mathcal{S}}f(s,g)\log(\frac{f_M(g)}{f(s)})ds \end{align} where $f_M(g) = \frac{1}{N}$ is the probability mass function for taking each $g$.

Using the law of total probability, for any $x$ in $\mathcal{S}$ and $y$ in $\mathcal{G}$: \begin{equation} f(x)=\sum_{\mathcal{G}}f(x|g)f_M(g). \end{equation} \begin{equation} f(y)=\sum_{\mathcal{G}}f(y|g)f_M(g)=\sum_{\mathcal{G}}\delta(y)f_M(g). \end{equation}

Then, since $f_M(g)$ is constant and can be cancelled, $f(y)/f(x)$ equals to \begin{equation} \frac{f(y)}{f(x)} = \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)}. \end{equation}

Form the property of Gaussian distribution we know that $f(x|g)$ is maximum when $x=g$, we estimate the lover bound of the above equation as: \begin{align} \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)} &\geq \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(\mu_g|g)} = \frac{\delta(y)}{Nf(\mu_g|g)} = \frac{+\infty}{Nf(\mu_g|g)} \nonumber \\ &= \frac{+\infty}{\frac{N}{\sigma\sqrt{2\pi}}} \end{align}

As $\mathbf{S}$ is Gaussian distributed around $g$, given $g$ is at the same location as $y$, for any given $x$ and $y$, $f(\mu_g|g)$ is always smaller than $+\infty$ and $\frac{f(y)}{f(x)}$ is always bigger than $1$. This means $\log(\frac{f(y)}{f(x)})$ is positive and $h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$ is positive as well. As a result, $\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$

Is the proof correct this time?

$\endgroup$
  • $\begingroup$ This is not true in general. Further, your proof strategy is off - you need to at least use some information about the spread of the $\mu_g$s. For example, suppose $\mu_g$ is identical for all $g,$ and $\sigma^2$ is small, then your expression is obviously negative. Now, by continuity of differential entropy, if I perturb the $\mu_g$ a little, so that they are all distinct, but still very close, the same should hold. How well spread the $\mu_g$ are has to enter your considerations, possibly as a condition for your desired inequality. $\endgroup$ – stochasticboy321 Aug 3 at 21:09
  • $\begingroup$ Very concretely - consider the following: let $G,Z$ be two independent standard Gaussians, and $S = G/K + Z$ for a given constant $K \gg1$. Then $h(S|G) = h(Z)$ but $h(G|S) = h(-KZ)$, and the latter is much larger (differential entropy in not invariant to scaling). Something similar should hold for discrete $Z$, although you will have to work harder to show it. $\endgroup$ – stochasticboy321 Aug 3 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.