We consider the two distributions $$ p_t = p_0 * N(0, tI),\quad q_t = q_0 * N(0, t I), $$ where $*$ denotes the convolution between two densities, while $p_0$ and $q_0$ have the same mean and variance. In particular, we assume that $q_0$ is $N(0, I)$. In other words, we consider two random variables $$ X_t = X_0 + N(0, t I)\sim p_t, \quad Y_t = Y_0 + N(0, tI)\sim q_t, $$ where $X_0\sim p_0$ and $Y_0\sim q_0 = N(0, I)$.

We are interested in characterizing the evolution of $$ \text{KL}(p_t, q_t) = \int p_t \log \left(\frac{p_t}{q_t}\right) dx $$ along $t$. In particular, we aim to prove:

(i) $\text{KL}(p_t, q_t)$ is monotonically decreasing along $t$;

(ii) $\text{KL}(p_t, q_t)$ is convex along t;

(iii) $\text{KL}(p_t, q_t)$ is smooth along t, that is, $\frac{d^2\text{KL}(p_t, q_t)}{dt^2} $ is upper bounded.

(Comment: As pointed out by Jon, (i) follows directly from the data processing inequality. As pointed out by Nawaf, (ii) also holds.)

  • 1
    $\begingroup$ Point (i) follows from the data processing inequality. $\endgroup$ – Jon Feb 16 at 0:38
  • $\begingroup$ @Jon You are absolutely right. Thanks a lot for pointing out. $\endgroup$ – Minkov Feb 16 at 3:33
  • 1
    $\begingroup$ Seems better to pose the new part (iii) as a new question, because it was added to the body and the title of the question after the OP received an answer; see, e.g., the nice discussion in meta.mathoverflow.net/questions/3057/splitting-a-question $\endgroup$ – Nawaf Bou-Rabee Feb 23 at 11:15
  • 1
    $\begingroup$ Minkov, I agree with Nawaf: he took the time to answer your question, and to change the question so as to render his answer now obsolete (or out-of-date as you put it) is not good form. I recommend that you accept his answer, and ask your part (iii) as a new question, with a bounty if you like. $\endgroup$ – Todd Trimble Feb 23 at 20:25
  • $\begingroup$ @NawafBou-Rabee You are absolutely right. I will take your answer and open a new question. Thanks again for your insightful answer! $\endgroup$ – Minkov Feb 23 at 22:01

Write the KL divergence in terms of the differential entropy of the random variables $X_t$ and $Y_t$; the result quickly follows. Indeed, since $Y_t \sim \mathcal{N}(0,1+t)$, we have \begin{align*} \operatorname{KL}(p_t, q_t) &= - h(p_t) + \frac{1}{2} \int \frac{x^2}{1+t} p_t(x) dx + \frac{1}{2} \log(1+t) + \frac{1}{2} \log(2 \pi) \\ &= - h(p_t) + \frac{1}{2} \mathbb{E}((X_0+\sqrt{t} \mathcal{N}(0,1))^2)\frac{1}{1+t} + \frac{1}{2} \log(1+t) + \frac{1}{2} \log(2 \pi) \\ &= -h(p_t) + h(q_t) \end{align*} where $h(\cdot)$ is the differential entropy. By Lemma 2 of Zhang, Anantharam and Geng, subject to $\operatorname{var}(X_0)=1$, the minimum of $-\frac{d^2}{dt^2} h(p_t)$ is achieved when $X_0$ is Gaussian. Thus, $-\frac{d^2}{dt^2} h(p_t) \ge -\frac{d^2}{dt^2} h(q_t)$, and hence, $\frac{d^2}{dt^2}\operatorname{KL}(p_t, q_t) \ge 0$ which implies that $\operatorname{KL}(p_t, q_t)$ is convex with respect to $t$.


The Gaussian minimality result used above seems to go back to

McKean, H. P., Speed of approach to equilibrium for Kac’s caricature of a Maxwellian gas, Arch. Ration. Mech. Anal. 21, 343-367 (1966). ZBL1302.60049.

  • 1
    $\begingroup$ Thanks a lot for the answer and the revision, Nawaf! $\endgroup$ – Minkov Feb 19 at 22:14
  • $\begingroup$ In fact, is it possible to prove that the second-order derivative is also upper bounded? $\endgroup$ – Minkov Feb 23 at 6:34
  • $\begingroup$ @Minkov The answer to the new part (iii) of this question seems to boil down to obtaining a bound on the second derivative of the differential entropy of $X_t$. A formula for this quantity -- attributed to H. McKean, G. Toscani, and C. Villani -- is given in (4) of mdpi.com/1099-4300/20/3/182/htm#B7-entropy-20-00182. This formula involves an expected value over $X_t$, and in order to bound, seems to require additional assumptions on $X_0$. $\endgroup$ – Nawaf Bou-Rabee Feb 23 at 11:11
  • $\begingroup$ I see. What kind of assumption is needed for this? One possible thing I have in mind is the lower boundedness of the density of $X_0$, which seems a bit strong. $\endgroup$ – Minkov Feb 23 at 22:23
  • $\begingroup$ @Minkov Agree that seems a bit strong; conjecture that the right assumptions are those that naturally come from expanding the expected value of McKean, Toscani and Villani in terms of the density of $X_0$, and then precisely estimating the terms. $\endgroup$ – Nawaf Bou-Rabee Feb 23 at 23:33

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.