Skip to main content
Bounty Ended with 50 reputation awarded by Maziar Sanjabi
added 71 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_0(t,s)}}+\frac{1-t}{f_{u_1(t,s)}}\Big)(1-s)\,ds, \end{align*} where $$u_j(t,s):=(1-s)t+js.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_0(t,s))+(1-t)I(u_1(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have $u_0(t,s)\in(0,1)$ and $u_1(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=0,1$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, assuming that $P_1$ is absolutely continuous w.r. to $P_0$ and introducing $f:=\frac{dP_1}{dP_0}$, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ areis as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_0(t,s)}}+\frac{1-t}{f_{u_1(t,s)}}\Big)(1-s)\,ds, \end{align*} where $$u_j(t,s):=(1-s)t+js.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_0(t,s))+(1-t)I(u_1(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have $u_0(t,s)\in(0,1)$ and $u_1(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=0,1$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_0(t,s)}}+\frac{1-t}{f_{u_1(t,s)}}\Big)(1-s)\,ds, \end{align*} where $$u_j(t,s):=(1-s)t+js.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_0(t,s))+(1-t)I(u_1(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have $u_0(t,s)\in(0,1)$ and $u_1(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=0,1$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, assuming that $P_1$ is absolutely continuous w.r. to $P_0$ and introducing $f:=\frac{dP_1}{dP_0}$, after some rather straightforward manipulations we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ is as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

deleted 22 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_1(t,s)}}+\frac{1-t}{f_{u_2(t,s)}}\Big)(1-s)\,ds, \end{align*}\begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_0(t,s)}}+\frac{1-t}{f_{u_1(t,s)}}\Big)(1-s)\,ds, \end{align*} where $$u_1(t,s):=(1-s)t,\quad u_2(t,s):=(1-s)t+s.$$$$u_j(t,s):=(1-s)t+js.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_1(t,s))+(1-t)I(u_2(t,s))], \tag{1} \end{equation}\begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_0(t,s))+(1-t)I(u_1(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have $u_1(t,s)\in(0,1)$$u_0(t,s)\in(0,1)$ and $u_2(t,s)\in(0,1)$$u_1(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=1,2$$j=0,1$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_1(t,s)}}+\frac{1-t}{f_{u_2(t,s)}}\Big)(1-s)\,ds, \end{align*} where $$u_1(t,s):=(1-s)t,\quad u_2(t,s):=(1-s)t+s.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_1(t,s))+(1-t)I(u_2(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have $u_1(t,s)\in(0,1)$ and $u_2(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=1,2$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_0(t,s)}}+\frac{1-t}{f_{u_1(t,s)}}\Big)(1-s)\,ds, \end{align*} where $$u_j(t,s):=(1-s)t+js.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_0(t,s))+(1-t)I(u_1(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have $u_0(t,s)\in(0,1)$ and $u_1(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=0,1$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

added 2 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_1(t,s)}}+\frac{1-t}{f_{u_2(t,s)}}\Big)(1-s)\,ds, \end{align*} where $u_1(t,s):=(1-s)t$ and $u_2(t,s):=(1-s)t+s$. $$u_1(t,s):=(1-s)t,\quad u_2(t,s):=(1-s)t+s.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_1(t,s))+(1-t)I(u_2(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have    $u_1(t,s)\in(0,1)$ and $u_2(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=1,2$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_1(t,s)}}+\frac{1-t}{f_{u_2(t,s)}}\Big)(1-s)\,ds, \end{align*} where $u_1(t,s):=(1-s)t$ and $u_2(t,s):=(1-s)t+s$. So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_1(t,s))+(1-t)I(u_2(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have  $u_1(t,s)\in(0,1)$ and $u_2(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=1,2$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

$\newcommand{\de}{\delta} \newcommand{\De}{\Delta} \newcommand{\ep}{\varepsilon}$

Take any probability measures $P_0,P_1$ absolutely continuous with respect (w.r.) to $Q$. We shall prove the following:

Theorem 1. For any $t\in(0,1)$, \begin{align*} \De:=(1-t)H(P_0)+tH(P_1)-H(P_t) \ge\frac{(1-t)t}2\,\|P_1-P_0\|^2, \end{align*} where $\|P_1-P_0\|:=\int|dP_1-dP_0|$ is the total variation norm of $P_1-P_0$, \begin{equation*} H(P):=D(P||Q)=\int \ln\frac{dP}{dQ}\,dP, \end{equation*} and, for any elements $C_0,C_1$ of a linear space, $C_t:=(1-t)C_0+tC_1$. Thus, by "A third definition[8] for a strongly convex function", indeed $D(P||Q)$ is strongly convex in $P$ w.r. to the total variation norm.

We see that the lower bound on $\De$ does not depend on $Q$.

Proof of Theorem 1. Take indeed any $t\in(0,1)$. Let $f_j:=\frac{dP_j}{dQ}$ for $j=0,1$, so that $f_t=\frac{dP_t}{dQ}$. By Taylor's theorem with the integral form of the remainder, for $h(x):=x\ln x$ and $j=0,1$ we have \begin{equation*} h(f_j)=h(f_t)+h'(f_t)(f_j-f_t)+(f_j-f_t)^2\int_0^1 h''((1-s)f_t+sf_j)(1-s)\,ds, \end{equation*} whence \begin{align*} \de&:=(1-t)h(f_0)+th(f_1)-h(f_t) \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{(1-s)f_t+sf_0}+\frac{1-t}{(1-s)f_t+sf_1}\Big)(1-s)\,ds \\ &=(1-t)t\,(f_1-f_0)^2\, \int_0^1\Big(\frac t{f_{u_1(t,s)}}+\frac{1-t}{f_{u_2(t,s)}}\Big)(1-s)\,ds, \end{align*} where $$u_1(t,s):=(1-s)t,\quad u_2(t,s):=(1-s)t+s.$$ So, \begin{equation} \De=\int\de\,dQ=(1-t)t\,\int_0^1(1-s)\,ds\,[tI(u_1(t,s))+(1-t)I(u_2(t,s))], \tag{1} \end{equation} where \begin{equation*} I(u):=\int\frac{(f_1-f_0)^2}{f_u}\,dQ. \end{equation*} Next, take any $u\in(0,1)$. Then $P_1$ is absolutely continuous w.r. to $P_u$. Introducing
$g_u:=\dfrac{dP_1}{dP_u}=\dfrac{f_1}{f_u}$, we have \begin{multline*} I(u)=\frac1{(1-u)^2}\,\int\frac{(f_1-f_u)^2}{f_u}\,dQ =\frac1{(1-u)^2}\,\int(g_u-1)^2\,dP_u \\ \ge\frac1{(1-u)^2}\,\Big(\int|g_u-1|\,dP_u\Big)^2 =\frac1{(1-u)^2}\,\|P_1-P_u\|^2=\|P_1-P_0\|^2. \tag{2} \end{multline*} Note also that for any $t\in(0,1)$ and $s\in(0,1)$ we have  $u_1(t,s)\in(0,1)$ and $u_2(t,s)\in(0,1)$ and hence, by (2), $I(u_j(t,s))\ge\|P_1-P_0\|^2$ for $j=1,2$. Now Theorem 1 follows by (1).

Remark. The constant factor $\frac12$ in the lower bound in Theorem 1 is the best possible one. Indeed, after some rather straightforward manipulations, we get \begin{equation} \De=\int k(t,f)\,dP_0, \tag{*} \end{equation} where $\De$ and $f=\frac{dP_1}{dP_0}$ are as before and \begin{equation} k(t,f):=t f \ln f-(1-t+t f)\ln(1-t+t f). \end{equation} Take now any $h\in(0,1)$ and let $f$ take values $1-h,1+h$ each on a set of $P_0$-measure $1/2$, so that $\|P_1-P_0\|=h$. Then, in view of (*), for each $t\in(0,1)$, \begin{equation} \De=\frac12\,k(t,1-h)+\frac12\,k(t,1+h)\sim \frac{(1-t)t}2\,h^2=\frac{(1-t)t}2\,\|P_1-P_0\|^2 \end{equation} as $h\downarrow0$, which confirms the optimality claim.

added 11 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
added 32 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
deleted 6 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
deleted 72 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
deleted 1720 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
added 1380 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
added 730 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
added 206 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
deleted 1 character in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
added 513 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
added 14 characters in body
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading
Source Link
Iosif Pinelis
  • 127.7k
  • 8
  • 107
  • 229
Loading