Given three distributions $p, q$ and $h$, assume we know that the Kullback-Leibler divergence obeys

  1. $KL(p\Vert q)$ is large enough, say $KL(p\Vert q) > M$ where $M$ is large enough, and

  2. $KL(q\Vert h)$ is small enough, $KL(q\Vert h) < t$ and $t$ is small enough (close to $0$).

Does there exist a number $N$ such that $KL(p\Vert h)>N$?

If yes, can someone give me a link to that conclusion?

If not, does there exist a 'metric' of the distance between distributions such that if $\operatorname{metric}(p,q)$ is large enough and $\operatorname{metric}(q,h)$ is small enough then $\operatorname{metric}(p,h)$ is larger than some value?

  • 2
    $\begingroup$ What is KL? (And more characters.) $\endgroup$
    – LSpice
    Dec 17, 2019 at 3:03

1 Answer 1


The answer to your main question, about the Kullback--Leibler divergence, is no. Indeed, let the vectors $p=(1-s,s)$, $q=(1-t,t)$, and $r=p$ represent the corresponding probability distributions on (say) the set $\{1,2\}$, where $s\downarrow0$ and $t:=e^{-1/s^2}$. Then $$KL(p||q)=(1-s)\ln\frac{1-s}{1-t}+s\ln\frac st\sim\frac1s\to\infty, $$ $$KL(q||r)=KL(q||p)=(1-t)\ln\frac{1-t}{1-s}+t\ln\frac ts\to0, $$ but $KL(p||r)=KL(p||p)=0$ -- which is of course not large but as small as $KL$ can be.

This phenomenon is general and not peculiar to distributions of a two-point set; it will occur on any nontrivial probability space. E.g., let the vectors $p=(\frac{1-s}{n-1},\dots,\frac{1-s}{n-1},s)$ and $q=(\frac{1-t}{n-1},\dots,\frac{1-t}{n-1},t)$ represent the corresponding probability distributions on the set $\{1,\dots,n\}$, for any natural $n\ge2$, where $s$ and $t$ are as above. Then we shall still have $KL(p||q)\to\infty$ but $KL(q||p)\to0$.

As for your "if not" question, there are many metrics on the set of all probability distributions. One of them is the Hellinger distance given by $$d_H(p,q)=\frac1{\sqrt2}\sqrt{\int(\sqrt p-\sqrt q)^2}. $$

For any such metric $d$ and any probability distributions $p,q,r$, by the triangle inequality we have $d(p,r)\ge d(p,q)-d(q,r)$. So, if $d(p,q)>M$ and $d(q,r)<t$, then $d(p,r)>M-t$.

  • $\begingroup$ Thank you very much. I see. The Bernoulli distribution becomes abnormal when $s$ approches 0. $\endgroup$ Dec 17, 2019 at 9:22
  • $\begingroup$ @user1388672 : This phenomenon will occur, not just with Bernouli distributions, but with distributions on any nontrivial probability space. I have added details on this. $\endgroup$ Dec 17, 2019 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.