Skip to main content
3 of 15
added 513 characters in body
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$ and to each other. We shall prove the following:

Theorem 1. For any $t\in[0,1]$, \begin{multline} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \\ \ge\frac{(1-t)t}2\,\Big[(1-t)\int\Big(\frac{dP_1}{dP_0}-1\Big)^2\,dP_0+t\int\Big(\frac{dP_0}{dP_1}-1\Big)^2\,dP_1\Big] \\ \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{multline} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound does not depend on $Q$.

Proof of Theorem 1. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation} whence, in view of Jensen's inequality for the convex function $[0,\infty)\ni x\to1/x$ and the probability measure $\mu(ds):=2(1-s)\,ds$ on $[0,1]$,
\begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=\frac{(1-t)t}2\,(f_1-f_0)^2\, \int_0^1\Big(\frac1{(1-s)f_t+sf_0}+\frac1{(1-s)f_t+sf_1}\Big)2(1-s)\,ds \\ &\ge\frac{(1-t)t}2\,(f_1-f_0)^2\,\Big(\frac1A+\frac1B\Big), \end{align*} where \begin{align*} A&:=\int_0^1[(1-s)f_t+sf_0]2(1-s)\,ds=f_{2t/3}, \\ B&:=\int_0^1[(1-s)f_t+sf_1]2(1-s)\,ds=f_{(2t+1)/3}. \end{align*} So, \begin{equation} \de\ge\frac{(1-t)t}2\,(f_1-f_0)^2\,u(t), \end{equation} where \begin{multline} u(t):=\frac1{f_{2t/3}}+\frac1{f_{(2t+1)/3}} \ge(1-t)u(0)+tu(1) \\ =(1-t)\Big(\frac1{f_0}+\frac1{f_{1/3}}\Big)+t\Big(\frac1{f_{2/3}}+\frac1{f_1}\Big) >\frac{1-t}{f_0}+\frac t{f_1}, \end{multline} since $u(t)$ is convex in $t$. So, \begin{multline} \De=\int\de\,dQ\ge\frac{(1-t)t}2\,\int(f_1-f_0)^2\,\Big(\frac{1-t}{f_0}+\frac t{f_1}\Big)dQ \\ =\frac{(1-t)t}2\,\Big[(1-t)\int\Big(\frac{dP_1}{dP_0}-1\Big)^2\,dP_0+t\int\Big(\frac{dP_0}{dP_1}-1\Big)^2\,dP_1\Big], \end{multline} which proves the first inequality in Theorem 1. The second inequality follows because, for $f:=\frac{dP_1}{dP_0}$, we have \begin{equation} \int\Big(\frac{dP_1}{dP_0}-1\Big)^2\,dP_0=\int(f-1)^2\,dP_0\ge\Big(\int|f-1|\,dP_0\Big)^2=\|P_1-P_0\|^2 \end{equation} and similarly \begin{equation} \int\Big(\frac{dP_0}{dP_1}-1\Big)^2\,dP_1\ge\|P_1-P_0\|^2. \end{equation} Theorem 1 is now completely proved.

Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229