MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top


$P_1$, $P_2$, $P_3$ are probability distributions defined on the same support.

Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ?

(H is the Shannon Entropy and $D_{KL}$ is the Kullback–Leibler divergence)

Thank you.

share|cite|improve this question
up vote 3 down vote accepted

In general there is no relation between the two divergences. In fact, both of the divergences may be either finite or infinite, independent of the values of the entropies.

To be precise, if $P_1$ is not absolutely continuous w.r.t. $P_2$, then $D_{KL}(P_2,P_1)=\infty$. Similarly, $D_{KL}(P_2,P_1)=\infty$. This fact is independent of the entropies of $P_1$, $P_2$ and $P_3$. Hence, by continuity, the ratio $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ can be arbitrary.

share|cite|improve this answer
Thank you. If we specify that KL is continuous at $(S_2, S_1)$ (respectively $(S_3, S_1)$) and that the distributions $S_1$, $S_2$, $S_3$ are strictly positive over all the support elements. Is it possible to characterize $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ ? – Raskol Apr 1 '13 at 15:54
Consider the following distributions on a state space of cardinality $n+2$: $P_1=((1-2\epsilon)Q_1,\epsilon,\epsilon)$, $P_2=((1-2\epsilon)Q_2,2\lambda\epsilon,2(1-\lambda)\epsilon)$, where $Q_1,Q_2$ are arbitrary distributions on $n$ states and $0<\lambda<1$. All three have full support, and for small $\epsilon$, $H(P_i)\approx H(Q_i)$. However, $D(P_2,P_1)=(1-4k\epsilon)D(Q_2,Q_1) + \epsilon\log(1/2\lambda) + \epsilon\log(1/2(1-\lambda))$. By choosing $\lambda$ conveniently, $D(P_2,P_1)$ can be tuned to any value between $(1-4k\epsilon)D(Q_2,Q_1)$ and $\infty$. – jarauh Apr 3 '13 at 12:33

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.