Relation between the two possible KL divergences of two distributions

Question

Given that I know $$D\left(P\parallel Q\right)<\alpha,$$ can I say anything about $D\left(Q\parallel P\right)$ in terms of an upper bound on it?

Also, given this upper bound on $D\left(P\parallel Q\right)$, can I deduce some upper bound on $\left|P\left(A\right)-Q\left(A\right)\right|$ for some arbitrary event $A$?

Answer 1

The answer to your first question is no: $D(Q||P)$ may be however large while $D(P||Q)$ is however small. E.g., let $P$ have masses $s$ and $1-s$ at points $0$ and $1$, respectively, and let $Q$ have masses $t$ and $1-t$ at points $0$ and $1$, respectively, where $0<s,t<1$. Then \begin{equation} D(P||Q)=s\ln\frac st+(1-s)\ln\frac{1-s}{1-t}, \end{equation} \begin{equation} D(Q||P)=t\ln\frac ts+(1-t)\ln\frac{1-t}{1-s}. \end{equation} Let now $t\downarrow0$ and $s=te^{-1/t^2}$. Then $D(P||Q)\to0$ whereas $D(Q||P)\to\infty$.

The answer to your second question is yes: Pinsker's inequality states that \begin{equation} \sup_A|P(A)-Q(A)|\le\sqrt{\tfrac12\,D(P||Q)}. \end{equation}