I’m having some problems in ensuring the non-negativity of KLD!
I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m getting negative results. Here is how I’m using KLD:
KLD( P(x) || Q(x) ) = Sum[ (P(x) . Log[P(x)/Q(x)] ]; Log is base 2. P(x) and Q(x) are two different distributions for all x belongs to X.
For example, P(x) = (Frequency(x)/Total_Size); just normal PMF! The same thing for Q(x). Note that the Total_Size for P might be different from Q distribution.
Could you please let me know if I’m missing something? Are there any special conditions that I have to take into consideration to avoid having negative results?
Thanks in advance.