$P_1$, $P_2$, $P_3$ are probability distributions defined on the same support.

Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ?

(H is the Shannon Entropy and $D_{KL}$ is the Kullback–Leibler divergence)

Thank you.


In general there is no relation between the two divergences. In fact, both of the divergences may be either finite or infinite, independent of the values of the entropies.

To be precise, if $P_1$ is not absolutely continuous w.r.t. $P_2$, then $D_{KL}(P_2,P_1)=\infty$. Similarly, $D_{KL}(P_2,P_1)=\infty$. This fact is independent of the entropies of $P_1$, $P_2$ and $P_3$. Hence, by continuity, the ratio $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ can be arbitrary.

  • $\begingroup$ Thank you. If we specify that KL is continuous at $(S_2, S_1)$ (respectively $(S_3, S_1)$) and that the distributions $S_1$, $S_2$, $S_3$ are strictly positive over all the support elements. Is it possible to characterize $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ ? $\endgroup$
    – Raskol
    Apr 1 '13 at 15:54
  • $\begingroup$ Consider the following distributions on a state space of cardinality $n+2$: $P_1=((1-2\epsilon)Q_1,\epsilon,\epsilon)$, $P_2=((1-2\epsilon)Q_2,2\lambda\epsilon,2(1-\lambda)\epsilon)$, where $Q_1,Q_2$ are arbitrary distributions on $n$ states and $0<\lambda<1$. All three have full support, and for small $\epsilon$, $H(P_i)\approx H(Q_i)$. However, $D(P_2,P_1)=(1-4k\epsilon)D(Q_2,Q_1) + \epsilon\log(1/2\lambda) + \epsilon\log(1/2(1-\lambda))$. By choosing $\lambda$ conveniently, $D(P_2,P_1)$ can be tuned to any value between $(1-4k\epsilon)D(Q_2,Q_1)$ and $\infty$. $\endgroup$
    – jarauh
    Apr 3 '13 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.