Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Hi,

$P_1$, $P_2$, $P_3$ are probability distributions defined on the same support.

Knowing that $H(P_1) < H(P_2) < H(P_3)$, can we compare $D_{KL}(P_2,P_1)$ and $D_{KL}(P_3,P_1)$ ?

(H is the Shannon Entropy and $D_{KL}$ is the Kullback–Leibler divergence)

Thank you.

share|improve this question

1 Answer 1

up vote 3 down vote accepted

In general there is no relation between the two divergences. In fact, both of the divergences may be either finite or infinite, independent of the values of the entropies.

To be precise, if $P_1$ is not absolutely continuous w.r.t. $P_2$, then $D_{KL}(P_2,P_1)=\infty$. Similarly, $D_{KL}(P_2,P_1)=\infty$. This fact is independent of the entropies of $P_1$, $P_2$ and $P_3$. Hence, by continuity, the ratio $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ can be arbitrary.

share|improve this answer
    
Thank you. If we specify that KL is continuous at $(S_2, S_1)$ (respectively $(S_3, S_1)) and that the distributions $S_1$, $S_2$, $S_3$ are strictly positive over all the support elements, is it possible to characterize $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ ? –  Raskol Apr 1 '13 at 15:52
    
Thank you. If we specify that KL is continuous at $(S_2, S_1)$ (respectively $(S_3, S_1)$) and that the distributions $S_1$, $S_2$, $S_3$ are strictly positive over all the support elements. Is it possible to characterize $D_{KL}(P_2,P_1)/D_{KL}(P_3,P_1)$ ? –  Raskol Apr 1 '13 at 15:54
    
Consider the following distributions on a state space of cardinality $n+2$: $P_1=((1-2\epsilon)Q_1,\epsilon,\epsilon)$, $P_2=((1-2\epsilon)Q_2,2\lambda\epsilon,2(1-\lambda)\epsilon)$, where $Q_1,Q_2$ are arbitrary distributions on $n$ states and $0<\lambda<1$. All three have full support, and for small $\epsilon$, $H(P_i)\approx H(Q_i)$. However, $D(P_2,P_1)=(1-4k\epsilon)D(Q_2,Q_1) + \epsilon\log(1/2\lambda) + \epsilon\log(1/2(1-\lambda))$. By choosing $\lambda$ conveniently, $D(P_2,P_1)$ can be tuned to any value between $(1-4k\epsilon)D(Q_2,Q_1)$ and $\infty$. –  jarauh Apr 3 '13 at 12:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.