# Given three distributions p, q and h. If KL(p||q) is large enough and KL(q||h) is small enough, does there exist a number N such that KL(p||h)>N?)

Given three distributions $$p, q$$ and $$h$$, assume we know that the Kullback-Leibler divergence obeys

1. $$KL(p\Vert q)$$ is large enough, say $$KL(p\Vert q) > M$$ where $$M$$ is large enough, and

2. $$KL(q\Vert h)$$ is small enough, $$KL(q\Vert h) < t$$ and $$t$$ is small enough (close to $$0$$).

Does there exist a number $$N$$ such that $$KL(p\Vert h)>N$$?

If yes, can someone give me a link to that conclusion?

If not, does there exist a 'metric' of the distance between distributions such that if $$\operatorname{metric}(p,q)$$ is large enough and $$\operatorname{metric}(q,h)$$ is small enough then $$\operatorname{metric}(p,h)$$ is larger than some value?

• What is KL? (And more characters.) Dec 17, 2019 at 3:03

The answer to your main question, about the Kullback--Leibler divergence, is no. Indeed, let the vectors $$p=(1-s,s)$$, $$q=(1-t,t)$$, and $$r=p$$ represent the corresponding probability distributions on (say) the set $$\{1,2\}$$, where $$s\downarrow0$$ and $$t:=e^{-1/s^2}$$. Then $$KL(p||q)=(1-s)\ln\frac{1-s}{1-t}+s\ln\frac st\sim\frac1s\to\infty,$$ $$KL(q||r)=KL(q||p)=(1-t)\ln\frac{1-t}{1-s}+t\ln\frac ts\to0,$$ but $$KL(p||r)=KL(p||p)=0$$ -- which is of course not large but as small as $$KL$$ can be.
This phenomenon is general and not peculiar to distributions of a two-point set; it will occur on any nontrivial probability space. E.g., let the vectors $$p=(\frac{1-s}{n-1},\dots,\frac{1-s}{n-1},s)$$ and $$q=(\frac{1-t}{n-1},\dots,\frac{1-t}{n-1},t)$$ represent the corresponding probability distributions on the set $$\{1,\dots,n\}$$, for any natural $$n\ge2$$, where $$s$$ and $$t$$ are as above. Then we shall still have $$KL(p||q)\to\infty$$ but $$KL(q||p)\to0$$.
As for your "if not" question, there are many metrics on the set of all probability distributions. One of them is the Hellinger distance given by $$d_H(p,q)=\frac1{\sqrt2}\sqrt{\int(\sqrt p-\sqrt q)^2}.$$
For any such metric $$d$$ and any probability distributions $$p,q,r$$, by the triangle inequality we have $$d(p,r)\ge d(p,q)-d(q,r)$$. So, if $$d(p,q)>M$$ and $$d(q,r), then $$d(p,r)>M-t$$.
• Thank you very much. I see. The Bernoulli distribution becomes abnormal when $s$ approches 0. Dec 17, 2019 at 9:22