Skip to main content
8 of 15
deleted 1720 characters in body
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. % and to each other. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bounds on $\De$ do not depend on $Q$.

Proof of Theorem 1. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence, in view of Jensen's inequality for the convex function $[0,\infty)\ni x\to1/x$ and the probability measure $\mu(ds):=2(1-s)\,ds$ on $[0,1]$,
\begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=\frac{(1-t)t}2\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)2(1-s)\,ds \\ &\ge\frac{(1-t)t}2\,(f_1-f_0)^2\,\Big(\frac tA+\frac{1-t}B\Big), \end{align*} where \begin{align*} A&:=\int_0^1[(1-s)f_t+sf_0]2(1-s)\,ds=f_{2t/3}, \\ B&:=\int_0^1[(1-s)f_t+sf_1]2(1-s)\,ds=f_{(2t+1)/3}. \end{align*} So, \begin{equation} \De=\int\de\,dQ\ge\frac{(1-t)t}2\,(tI_1+(1-t)I_2), \tag{1} \end{equation} where \begin{equation*} I_1:=\int\frac{(f_1-f_0)^2}{f_{2t/3}}\,dQ,\quad I_2:=\int\frac{(f_1-f_0)^2}{f_{(2t+1)/3}}\,dQ. \end{equation*} Next, introducing $g:=\frac{dP_1}{dP_{2t/3}}=\frac{f_1}{f_{2t/3}}$, we have \begin{multline*} I_1=\frac1{(1-2t/3)^2}\,\int\frac{(f_1-f_{2t/3})^2}{f_{2t/3}}\,dQ =\frac1{(1-2t/3)^2}\,\int(g-1)^2\,dP_{2t/3} \\ \ge\frac1{(1-2t/3)^2}\,\Big(\int|g-1|\,dP_{2t/3}\Big)^2 =\frac1{(1-2t/3)^2}\,\|P_1-P_{2t/3}\|^2=\|P_1-P_0\|^2, \end{multline*} so that $I_1\ge\|P_1-P_0\|^2$. Quite similarly, $I_2\ge\|P_1-P_0\|^2$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the first lower bound in Theorem 1 is the best possible one. %, at least for small values of $\|P_1-P_0\|$. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229