I’m having some problems in ensuring the non-negativity of KLD!
I know that KLD is always positive and I went over the proof. However, it doesn’t seem to work for me. In some cases I’m getting negative results. Here is how I’m using KLD:
$${\rm KLD}( P(x) || Q(x) ) = \sum P(x) \log \left( \frac{P(x)}{Q(x)} \right) \, ,$$ where the Log is in base 2, and $P(x)$ and $Q(x)$ are two different distributions for all $x \in X$.
For example, $P(x) = {\rm Frequency}(x)/{\text Total Size}$; just a normal PMF! The same thing for $Q(x)$. Note that the Total_Size of $P$ might be different from that of the $Q$ distribution.
Could you please let me know if I’m missing something? Are there any special conditions that I have to take into consideration to avoid having negative results?