Skip to main content
1 of 15
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\al}{\alpha} \newcommand{\be}{\beta} \newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon} \newcommand{\ga}{\gamma} \newcommand{\Ga}{\Gamma} \newcommand{\la}{\lambda} \newcommand{\si}{\sigma} \newcommand{\Si}{\Sigma} \newcommand{\thh}{\theta} \newcommand{\om}{\omega} \newcommand{\R}{\mathbb{R}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\F}{\mathcal{F}} \newcommand{\E}{\operatorname{\mathsf E}} \newcommand{\Var}{\operatorname{\mathsf Var}} \renewcommand{\P}{\operatorname{\mathsf P}} \newcommand{\ii}[1]{\operatorname{\mathsf I}\{#1\}} \newcommand{\eD}{\overset{\text{D}}\to} \newcommand{\D}{\overset{\text{D}}=} \newcommand{\tsi}{\tilde\si}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect to $Q$ and to each other. We shall prove the following:

Theorem 1. For any $t\in[0,1]$, \begin{multline} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \\ \ge\frac{(1-t)t}2\,\Big[(1-t)\int\Big(\frac{dP_1}{dP_0}-1\Big)^2\,dP_0+t\int\Big(\frac{dP_0}{dP_1}-1\Big)^2\,dP_1\Big], \end{multline} where \begin{equation} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP \end{equation} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$.

We see that the lower bound does not depend on $Q$.

Proof of Theorem 1. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation} whence, in view of Jensen's inequality for the convex function $[0,\infty)\ni x\to1/x$ and the probability measure $2(1-s)\,ds$ on $[0,1]$,
\begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=\frac{(1-t)t}2\,(f_1-f_0)^2\, \int_0^1\Big(\frac1{(1-s)f_t+sf_0}+\frac1{(1-s)f_t+sf_1}\Big)2(1-s)\,ds \\ &\ge\frac{(1-t)t}2\,(f_1-f_0)^2\,\Big(\frac1A+\frac1B\Big), \end{align*} where \begin{align*} A&:=\int_0^1[(1-s)f_t+sf_0]2(1-s)\,ds=f_{2t/3}, \\ B&:=\int_0^1[(1-s)f_t+sf_1]2(1-s)\,ds=f_{(2t+1)/3}. \end{align*} So, \begin{equation} \de\ge\frac{(1-t)t}2\,(f_1-f_0)^2\,u(t), \end{equation} where \begin{multline} u(t):=\frac1{f_{2t/3}}+\frac1{f_{(2t+1)/3}} \ge(1-t)u(0)+tu(1) \\ =(1-t)\Big(\frac1{f_0}+\frac1{f_{1/3}}\Big)+t\Big(\frac1{f_{2/3}}+\frac1{f_1}\Big) >\frac{1-t}{f_0}+\frac t{f_1}, \end{multline} since $u(t)$ is convex in $t$. So, \begin{multline} \De=\int\de\,dQ\ge\frac{(1-t)t}2\,\int(f_1-f_0)^2\,\Big(\frac{1-t}{f_0}+\frac t{f_1}\Big)dQ \\ =\frac{(1-t)t}2\,\Big[(1-t)\int\Big(\frac{dP_1}{dP_0}-1\Big)^2\,dP_0+t\int\Big(\frac{dP_0}{dP_1}-1\Big)^2\,dP_1\Big]. \Box \end{multline}

Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229