# KL divergence(s) comparison,

Hi,

$P_1$, $P_2$, $P_3$ are probability distributions defined on the same support.

Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ?

(H is the Shannon Entropy and $D_{KL}$ is the Kullback–Leibler divergence)

Thank you.

-

To be precise, if $P_1$ is not absolutely continuous w.r.t. $P_2$, then $D_{KL}(P_2,P_1)=\infty$. Similarly, $D_{KL}(P_2,P_1)=\infty$. This fact is independent of the entropies of $P_1$, $P_2$ and $P_3$. Hence, by continuity, the ratio $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ can be arbitrary.
Thank you. If we specify that KL is continuous at $(S_2, S_1)$ (respectively $(S_3, S_1)) and that the distributions$S_1$,$S_2$,$S_3$are strictly positive over all the support elements, is it possible to characterize$D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$? – Raskol Apr 1 '13 at 15:52 Thank you. If we specify that KL is continuous at$(S_2, S_1)$(respectively$(S_3, S_1)$) and that the distributions$S_1$,$S_2$,$S_3$are strictly positive over all the support elements. Is it possible to characterize$D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$? – Raskol Apr 1 '13 at 15:54 Consider the following distributions on a state space of cardinality$n+2$:$P_1=((1-2\epsilon)Q_1,\epsilon,\epsilon)$,$P_2=((1-2\epsilon)Q_2,2\lambda\epsilon,2(1-\lambda)\epsilon)$, where$Q_1,Q_2$are arbitrary distributions on$n$states and$0<\lambda<1$. All three have full support, and for small$\epsilon$,$H(P_i)\approx H(Q_i)$. However,$D(P_2,P_1)=(1-4k\epsilon)D(Q_2,Q_1) + \epsilon\log(1/2\lambda) + \epsilon\log(1/2(1-\lambda))$. By choosing$\lambda$conveniently,$D(P_2,P_1)$can be tuned to any value between$(1-4k\epsilon)D(Q_2,Q_1)$and$\infty\$. –  jarauh Apr 3 '13 at 12:33