Skip to main content
7 of 15
added 1380 characters in body
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$ and to each other. We shall prove the following:

Theorem 1. For any $t\in[0,1]$, \begin{align*} \De&:=(1-t)H(P_0)+tH(P_1)-H(P_t) \\ %&\ge\frac{(1-t)t}2\,\Big[(1-t)\int\Big(\frac{dP_1}{dP_0}-1\Big)^2\,dP_0+t\int\Big(\frac{dP_0}{dP_1}-1\Big)^2\,dP_1\Big] \\ & \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2(1-\|P_1-P_0\|/2) \ge\frac{(1-t)t}4\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bounds on $\De$ do not depend on $Q$.

Proof of Theorem 1. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence, in view of Jensen's inequality for the convex function $[0,\infty)\ni x\to1/x$ and the probability measure $\mu(ds):=2(1-s)\,ds$ on $[0,1]$,
\begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=\frac{(1-t)t}2\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)2(1-s)\,ds \\ &\ge\frac{(1-t)t}2\,(f_1-f_0)^2\,\Big(\frac tA+\frac{1-t}B\Big), \end{align*} where \begin{align*} A&:=\int_0^1[(1-s)f_t+sf_0]2(1-s)\,ds=f_{2t/3}, \\ B&:=\int_0^1[(1-s)f_t+sf_1]2(1-s)\,ds=f_{(2t+1)/3}. \end{align*} So, \begin{equation*} \de\ge\frac{(1-t)t}2\,I(t,f)f_0, %(f_1-f_0)^2\,u(t), \tag{1} \end{equation*} where \begin{equation*} f:=\frac{f_1}{f_0}=\frac{dP_1}{dP_0}, \end{equation*} \begin{multline*} I(t,f):=\frac1{f_0}\,(f_1-f_0)^2\,\Big(\frac t{f_{2t/3}}+\frac{1-t}{f_{(2t+1)/3}}\Big) =\frac{9 (f-1)^2 ((f-1) t+1)}{(2 (f-1) t+3) (2 f t+f-2 t+2)} \\ \ge J(f):=\frac{3(f-1)^2}{f+1+f\vee1}. % u(t):=\frac1{f_{2t/3}}+\frac1{f_{(2t+1)/3}} % \ge(1-t)u(0)+tu(1) \\ % =(1-t)\Big(\frac1{f_0}+\frac1{f_{1/3}}\Big)+t\Big(\frac1{f_{2/3}}+\frac1{f_1}\Big) % >\frac{1-t}{f_0}+\frac t{f_1}, \tag{2} \end{multline*} Since $J''(f):=\frac{54}{(f+1+f\vee1)^3}>0$, the function $J$ is convex. So, for any real $c>0$ we have $J(f)\ge J(c)+J'(c)(f-c)$. Take now any $u\in(0,1)$ and then \begin{equation} v:=\frac{4-u}{1+2u}.\tag{3} \end{equation} Then $v>1$, $$a(u):=J(u)+J'(u)(1-u)=-\frac{9 (1-u)^2}{(2 + u)^2}=a(v),$$ and \begin{multline*} J(f)\ge[J(u)+J'(u)(1-u)]\vee[J(v)+J'(v)(1-v)] \\ =a(u)+\frac{J'(u)+J'(v)}2\,(f-1)+\frac{J'(v)-J'(u)}2\,|f-1|. \end{multline*} Also, $\int f\,dP_0=1=\int dP_0$ and $$\int|f-1|\,dP_0=\|P_1-P_0\|=:\ep.$$ So, \begin{equation*} \int J(f)\,dP_0\ge a(u)+\frac{J'(v)-J'(u)}2\,\ep=:K(\ep,u), \end{equation*} with $v$ as in (3). So, in view of (1) and (2), \begin{multline*} \De=\int\de\,dQ \ge\frac{(1-t)t}2\,\int I(t,f)f_0\,dQ =\frac{(1-t)t}2\,\int I(t,f)\,dP_0 \\ \ge\frac{(1-t)t}2\,\int J(f)\,dP_0 \ge\frac{(1-t)t}2\,K(\ep,u). \end{multline*} Note that $K(\ep,u)$ is rational in $u$. The maximum, say $m(\ep)$, of $K(\ep,u)$ in $u\in(0,1)$ is a root of a polynomial (of degree $4$) whose coefficients are polynomials in $\ep$. Using then a computer algebra package, one verifies that $m(\ep)\ge\ep^2(1-\ep/2)$; details of calculations can be found in the Mathematica notebook or its pdf image. Thus, $\De\ge\frac{(1-t)t}2\,\ep^2(1-\ep/2)$; that is, the first inequality in Theorem 1 is proved. The second inequality there is trivial.

Theorem 1 is now completely proved.

Remark. The constant factor $\frac12$ in the first lower bound in Theorem 1 is the best possible one, at least for small values of $\|P_1-P_0\|$. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229