Given three distributions p, q and h. If KL(p||q) is large enough and KL(q||h) is small enough, does there exist a number N such that KL(p||h)>N?)

Question

Given three distributions $p, q$ and $h$, assume we know that the Kullback-Leibler divergence obeys

1. $KL(p\Vert q)$ is large enough, say $KL(p\Vert q) > M$ where $M$ is large enough, and

2. $KL(q\Vert h)$ is small enough, $KL(q\Vert h) < t$ and $t$ is small enough (close to $0$).

Does there exist a number $N$ such that $KL(p\Vert h)>N$?

If yes, can someone give me a link to that conclusion?

If not, does there exist a 'metric' of the distance between distributions such that if $\operatorname{metric}(p,q)$ is large enough and $\operatorname{metric}(q,h)$ is small enough then $\operatorname{metric}(p,h)$ is larger than some value?

Answer 1

The answer to your main question, about the Kullback--Leibler divergence, is no. Indeed, let the vectors $p=(1-s,s)$, $q=(1-t,t)$, and $r=p$ represent the corresponding probability distributions on (say) the set $\{1,2\}$, where $s\downarrow0$ and $t:=e^{-1/s^2}$. Then $$KL(p||q)=(1-s)\ln\frac{1-s}{1-t}+s\ln\frac st\sim\frac1s\to\infty, $$ $$KL(q||r)=KL(q||p)=(1-t)\ln\frac{1-t}{1-s}+t\ln\frac ts\to0, $$ but $KL(p||r)=KL(p||p)=0$ -- which is of course not large but as small as $KL$ can be.

---

This phenomenon is general and not peculiar to distributions of a two-point set; it will occur on any nontrivial probability space. E.g., let the vectors $p=(\frac{1-s}{n-1},\dots,\frac{1-s}{n-1},s)$ and $q=(\frac{1-t}{n-1},\dots,\frac{1-t}{n-1},t)$ represent the corresponding probability distributions on the set $\{1,\dots,n\}$, for any natural $n\ge2$, where $s$ and $t$ are as above. Then we shall still have $KL(p||q)\to\infty$ but $KL(q||p)\to0$.

---

As for your "if not" question, there are many metrics on the set of all probability distributions. One of them is the Hellinger distance given by $$d_H(p,q)=\frac1{\sqrt2}\sqrt{\int(\sqrt p-\sqrt q)^2}. $$

For any such metric $d$ and any probability distributions $p,q,r$, by the triangle inequality we have $d(p,r)\ge d(p,q)-d(q,r)$. So, if $d(p,q)>M$ and $d(q,r)<t$, then $d(p,r)>M-t$.