Suppose that $D_{KL}(p_1\parallel q)<1$ and $D_{KL}(p_2\parallel q)<1$. I'm trying to show that either $D_{KL}(p_1\parallel p_2)$ or $D_{KL}(p_2\parallel p_1)$ will have an upper bound close to 1 provided ${q}$ is fixed. It seems intuitive that if $p_1$ and $q$ are similar enough and if $p_2$ and $q$ are also similar enough then $p_2$ can reasonably approximate $p_1$ or vice versa. Is this actually true?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ What do you mean by "close to 1"? $\endgroup$– Iosif PinelisCommented Jul 28, 2022 at 21:50
-
1$\begingroup$ @losif Pinelis, I mean the bound should be finite and small $\endgroup$– Harry LorentzCommented Jul 28, 2022 at 22:21
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The answer is no. E.g., let $p_1,p_2,q$ be pdf's on $[0,1]$ such that $q=1$, $p_1=\frac1{1-h}\,1_{[0,1-h]}$, and $p_2=\frac1{1-h}\,1_{[h,1]}$, for some $h\in(0,1)$. Then $$D_{KL}(p_1\parallel q)=D_{KL}(p_2\parallel q)=\ln\frac1{1-h}\to0$$ as $h\downarrow0$, whereas $$D_{KL}(p_1\parallel p_2)=D_{KL}(p_2\parallel p_1)=\infty$$ for all $h\in(0,1)$.
answered Jul 28, 2022 at 22:36
-
1$\begingroup$ This is a nice counterexample, which emphasizes that KL is neither symmetric nor does it satisfy the triangle inequality. Nonetheless, sometimes for special classes of distributions, a weak form of the triangle inequality holds (see, e.g., arxiv.org/pdf/2102.05485.pdf ). $\endgroup$ Commented Jul 29, 2022 at 11:25
-
$\begingroup$ @NawafBou-Rabee : Thank you for the reference. $\endgroup$ Commented Jul 29, 2022 at 12:52