# On the difference of conditional differential entropy of two correlated random variables

## Problem Definition

Let $$\mathbf{G}$$ and $$\mathbf{S}$$ be jointly distributed random variables where $$\mathbf{S}$$ is continuous and is related to $$\mathbf{G}$$ through a conditional pdf $$f(s|g)$$ defined for all g. The conditional differential entropy of $$\mathbf{S}$$ given $$\mathbf{G}$$ is defined as \begin{align}\label{hsg} h(\mathbf{S}|\mathbf{G}) &= -\int_{\mathcal{G,S}}f(s,g)\log(f(s|g))dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg. \end{align} If $$f(\mathbf{S}|\mathbf{G}=g)$$ is a Gaussian distribution centred at $$g$$, we want to proof the following expression is true: \begin{align} \label{definition_s_g} \color{blue} { h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0 } \end{align}

This is our idea to prove it, could someone check if our proof is correct?

## Our Proof

Using the above conditional entropy definition, $$h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$$ can be rewritten as: \begin{align} h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)})dsdg \nonumber \\ &- \int_{\mathcal{G,S}}f(s,g)\log(\frac{p(s)}{f(s,g)})dsdg \nonumber \\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s,g)} \frac{f(s,g)}{f(s)})dsdg \nonumber\\ &= \int_{\mathcal{G,S}}f(s,g)\log(\frac{f(g)}{f(s)})dsdg. \end{align}

If we want to prove $$h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S})$$ is positive, we can prove that $$p(g)/p(s)$$ is greater than $$1$$. Since $$f(s,g)$$ is always positive, if we can prove that $$log(\frac{f(g)}{f(s)})$$ is also positive for any given $$s$$ and $$g$$, we can say the integral is also positive.

The likelihood of any $$x\in \mathcal{S}$$ can be computed by $$$$f(x)=\int_{\mathcal{G}}f(\mathbf{S}=x|\mathbf{G}=g)f(\mathbf{G}=g)dg.$$$$ For any $$y\in \mathcal{G}$$, we can write $$p(y)$$ in the same way as $$$$f(y)=\int_{\mathcal{G}}f(\mathbf{G}=y|\mathbf{G}=g)f(\mathbf{G}=g)dg.$$$$ Then, $$f(y)/f(x)$$ equals to $$$$\frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg. \label{equ:py_ps}$$$$

Since $$f(y|g)=0$$ when $$g\neq y$$, we can further simplify the equation as: $$$$\frac{f(y)}{f(x)} = \frac{f(y|g=y)}{f(x|g=y)} = \frac{\delta(y)}{f(x|g=y)} = \frac{+\infty}{f(x|g=y)},$$$$ where $$\delta(\cdot)$$ is a Dirac delta function.

As $$\mathbf{S}$$ is Gaussian distributed around $$g$$, given $$g=y$$, for any given $$x \in \mathbf{S}$$ and $$y \in \mathbf{G}$$, $$f(x|g=y)$$ is always smaller than $$+\infty$$, then $$\frac{p(y)}{p(x)}$$ is always bigger than $$1$$. This means $$\log(\frac{p(y)}{p(x)})$$ is positive and $$h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$$ is positive as well. As a result, $$\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$$

## Update 1

Now I see this proof is wrong because integral does not distribute over fraction. So we cannot get $$$$\frac{f(y)}{f(x)} = \int_{\mathcal{G}}\frac{f(y|g)}{f(x|g)}dg.$$$$

Are there other solutions?

## Update 2: Modification on the problem definition

In order to simplify the problem we give additional constrains on \mathbf{G}:

• $$\mathbf{G}$$ takes N possible values and follows a discrete uniform distribution

The rests are the same, $$\mathbf{S}$$ is continuous and is related to $$\mathbf{G}$$ through a conditional pdf $$f(\mathbf{S}|\mathbf{G}=g)=\mathcal{N}(\mu_g, \sigma^2)$$ centred at different $$g$$.

The proves are: \begin{align} h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) =\sum_{\mathcal{G}} \int_{\mathcal{S}}f(s,g)\log(\frac{f_M(g)}{f(s)})ds \end{align} where $$f_M(g) = \frac{1}{N}$$ is the probability mass function for taking each $$g$$.

Using the law of total probability, for any $$x$$ in $$\mathcal{S}$$ and $$y$$ in $$\mathcal{G}$$: $$$$f(x)=\sum_{\mathcal{G}}f(x|g)f_M(g).$$$$ $$$$f(y)=\sum_{\mathcal{G}}f(y|g)f_M(g)=\sum_{\mathcal{G}}\delta(y)f_M(g).$$$$

Then, since $$f_M(g)$$ is constant and can be cancelled, $$f(y)/f(x)$$ equals to $$$$\frac{f(y)}{f(x)} = \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)}.$$$$

Form the property of Gaussian distribution we know that $$f(x|g)$$ is maximum when $$x=g$$, we estimate the lover bound of the above equation as: \begin{align} \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(x|g)} &\geq \frac{\sum_{\mathcal{G}}\delta(y)}{\sum_{\mathcal{G}}f(\mu_g|g)} = \frac{\delta(y)}{Nf(\mu_g|g)} = \frac{+\infty}{Nf(\mu_g|g)} \nonumber \\ &= \frac{+\infty}{\frac{N}{\sigma\sqrt{2\pi}}} \end{align}

As $$\mathbf{S}$$ is Gaussian distributed around $$g$$, given $$g$$ is at the same location as $$y$$, for any given $$x$$ and $$y$$, $$f(\mu_g|g)$$ is always smaller than $$+\infty$$ and $$\frac{f(y)}{f(x)}$$ is always bigger than $$1$$. This means $$\log(\frac{f(y)}{f(x)})$$ is positive and $$h(\mathbf{S}|\mathbf{G}) - h(\mathbf{S}|\mathbf{G})$$ is positive as well. As a result, $$\color{blue}{h(\mathbf{S}|\mathbf{G}) - h(\mathbf{G}|\mathbf{S}) > 0}$$

Is the proof correct this time?

• This is not true in general. Further, your proof strategy is off - you need to at least use some information about the spread of the $\mu_g$s. For example, suppose $\mu_g$ is identical for all $g,$ and $\sigma^2$ is small, then your expression is obviously negative. Now, by continuity of differential entropy, if I perturb the $\mu_g$ a little, so that they are all distinct, but still very close, the same should hold. How well spread the $\mu_g$ are has to enter your considerations, possibly as a condition for your desired inequality. – stochasticboy321 Aug 3 at 21:09
• Very concretely - consider the following: let $G,Z$ be two independent standard Gaussians, and $S = G/K + Z$ for a given constant $K \gg1$. Then $h(S|G) = h(Z)$ but $h(G|S) = h(-KZ)$, and the latter is much larger (differential entropy in not invariant to scaling). Something similar should hold for discrete $Z$, although you will have to work harder to show it. – stochasticboy321 Aug 3 at 21:09