# KL divergence(s) comparison,

Hi,

$P_1$, $P_2$, $P_3$ are probability distributions defined on the same support.

Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ?

(H is the Shannon Entropy and $D_{KL}$ is the Kullback–Leibler divergence)

Thank you.

-

To be precise, if $P_1$ is not absolutely continuous w.r.t. $P_2$, then $D_{KL}(P_2,P_1)=\infty$. Similarly, $D_{KL}(P_2,P_1)=\infty$. This fact is independent of the entropies of $P_1$, $P_2$ and $P_3$. Hence, by continuity, the ratio $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ can be arbitrary.
Thank you. If we specify that KL is continuous at $(S_2, S_1)$ (respectively $(S_3, S_1)$) and that the distributions $S_1$, $S_2$, $S_3$ are strictly positive over all the support elements. Is it possible to characterize $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ ? – Raskol Apr 1 '13 at 15:54
Consider the following distributions on a state space of cardinality $n+2$: $P_1=((1-2\epsilon)Q_1,\epsilon,\epsilon)$, $P_2=((1-2\epsilon)Q_2,2\lambda\epsilon,2(1-\lambda)\epsilon)$, where $Q_1,Q_2$ are arbitrary distributions on $n$ states and $0<\lambda<1$. All three have full support, and for small $\epsilon$, $H(P_i)\approx H(Q_i)$. However, $D(P_2,P_1)=(1-4k\epsilon)D(Q_2,Q_1) + \epsilon\log(1/2\lambda) + \epsilon\log(1/2(1-\lambda))$. By choosing $\lambda$ conveniently, $D(P_2,P_1)$ can be tuned to any value between $(1-4k\epsilon)D(Q_2,Q_1)$ and $\infty$. – jarauh Apr 3 '13 at 12:33