3
$\begingroup$

Suppose that $D_{KL}(p_1\parallel q)<1$ and $D_{KL}(p_2\parallel q)<1$. I'm trying to show that either $D_{KL}(p_1\parallel p_2)$ or $D_{KL}(p_2\parallel p_1)$ will have an upper bound close to 1 provided ${q}$ is fixed. It seems intuitive that if $p_1$ and $q$ are similar enough and if $p_2$ and $q$ are also similar enough then $p_2$ can reasonably approximate $p_1$ or vice versa. Is this actually true?

$\endgroup$
2
  • 1
    $\begingroup$ What do you mean by "close to 1"? $\endgroup$ Jul 28, 2022 at 21:50
  • 1
    $\begingroup$ @losif Pinelis, I mean the bound should be finite and small $\endgroup$ Jul 28, 2022 at 22:21

1 Answer 1

4
$\begingroup$

The answer is no. E.g., let $p_1,p_2,q$ be pdf's on $[0,1]$ such that $q=1$, $p_1=\frac1{1-h}\,1_{[0,1-h]}$, and $p_2=\frac1{1-h}\,1_{[h,1]}$, for some $h\in(0,1)$. Then $$D_{KL}(p_1\parallel q)=D_{KL}(p_2\parallel q)=\ln\frac1{1-h}\to0$$ as $h\downarrow0$, whereas $$D_{KL}(p_1\parallel p_2)=D_{KL}(p_2\parallel p_1)=\infty$$ for all $h\in(0,1)$.

$\endgroup$
2
  • 1
    $\begingroup$ This is a nice counterexample, which emphasizes that KL is neither symmetric nor does it satisfy the triangle inequality. Nonetheless, sometimes for special classes of distributions, a weak form of the triangle inequality holds (see, e.g., arxiv.org/pdf/2102.05485.pdf ). $\endgroup$ Jul 29, 2022 at 11:25
  • $\begingroup$ @NawafBou-Rabee : Thank you for the reference. $\endgroup$ Jul 29, 2022 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.