Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Hello,

I’m having some problems in ensuring the non-negativity of KLD!

I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m getting negative results. Here is how I’m using KLD:

KLD( P(x) || Q(x) ) = Sum[ (P(x) . Log[P(x)/Q(x)] ]; Log is base 2. P(x) and Q(x) are two different distributions for all x belongs to X.

For example, P(x) = (Frequency(x)/Total_Size); just normal PMF! The same thing for Q(x). Note that the Total_Size for P might be different from Q distribution.

Could you please let me know if I’m missing something? Are there any special conditions that I have to take into consideration to avoid having negative results?

Thanks in advance.

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

"The K-L divergence is only defined if P and Q both sum to 1 and if Q(i) > 0 for any i such that P(i) > 0."

I suspect that the second condition is your problem. Say that you have x which appears in P but not Q -- in this case you're probably adding zero contribution to the sum in your code so that you don't have to divide by zero or take the logarithm of zero, but this is effectively throwing out mass from P and you get a negative number for the divergence.

http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence

share|improve this answer
    
Thank you Ohai! –  Mike Oct 27 '10 at 19:32
add comment

Here is a simple proof, which also shows non-negativity for several other related functions.

Let $\varphi : \mathbb{R}^n \to \mathbb{R}$ be a strictly convex and continuously differentiable function. Then we can use $\varphi$ to "generate" a divergence function $D$ defined as follows: $$D(x,y) = \phi(x) - \phi(y) - \langle \nabla\varphi(y), x - y\rangle$$

The above divergence function is known as a Bregman divergence.

The fact that $D(x,y) \ge 0$ (equalling $0$, iff $x=y$) is an immediate consequence of the strict convexity of $\varphi$.

Now, if you choose $\varphi(x) = \sum_i x_i \log x_i$, you obtain the KL-Divergence (upto normalization) --- Note here that usually one takes $0\log 0 = 0$ (by continuity), and that the arguments to KL should satisfy the conditions mentioned by Ohai.

share|improve this answer
    
Thank you Suvrit! –  Mike Oct 27 '10 at 19:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.