Assume I have a continuos random variable $X$, whose support is all $\mathbb R$. Let $Z$ be a standard normal independent on $X$, and let

$$Y = X + \sigma Z$$

$Y$ essentially is equal to $X$ plus "random noise". Clearly, its pdf is $f_Y = (f_X * \varphi(x, \sigma))(x)$ where $\varphi(x, \sigma)$ is the pdf of $\sigma Z$.

Let $$D_{KL}(\sigma) = D_{KL}(X \mid Y) = \int_\mathbb R f_X(x) \log\frac{f_X(x)}{f_Y(x)}dx$$

be the Kullback-Leibler divergence between $X$ and $Y$. It's immediate to see that $D_{KL}(\sigma) > 0$ for $\sigma > 0$ and $D_{KL}(0) = 0$. I would like to prove that $D_{KL}(\sigma)$ is increasing with $\sigma$. This seems reasonable; the KL divergence is a measure of how "divergent" the two distributions are, and with bigger $\sigma$ we add more noise, hence I would expect the KL divergence to increase. I am struggling to prove this, do you have any ideas?


Some numerical analysis suggests that is true (I tried when $X$ is a normal, a student t, and a member of the stable distribution for different combination of parameters). These analysis even suggest that $D_{KL}(\sigma)$ is convex in $\sigma$.

I can't seem to prove it though. I tried to use the data processing inequality and the chain rule for kl divergence (the formula that uses the conditional kl divergence) but with little luck.

Note that a straight-forward differentiation wrt to $\sigma$ yields

$$\frac{\partial D_{KL}(\sigma)}{\partial \sigma} = -\sigma \int_\mathbb R f_X(x) \frac{f_Y''(x, \sigma)}{f_Y(x, \sigma)}dx$$

But proving that this term is positive is not so easy ($'$ indicates the derivative wrt to $x$) It may help to notice that $\int_\mathbb R f''_Y(x, \sigma) dx = 0$, but I haven't really made any progress with this either.

Thank you!

  • $\begingroup$ Have you tried to find a counter-example (perhaps numerically)? X with two length scale mixture with two or three modes might show something interesting? Intermediate amount of smoothing shows less (KL) similarity seems plausible. $\endgroup$ – Memming Oct 19 '17 at 21:10
  • $\begingroup$ @Memming Uhm, good points. I will try that. But maybe is it true for sigma "small enough"? With sigma small enough we shouldn't observe this phenomenon :) $\endgroup$ – Ant Oct 20 '17 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.