MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

2 added 362 characters in body

I'll give a short answer and a long answer.

The short answer is that the KL divergence on multinomials is defined when they have only nonzero entries. When there are zero entries, you have two choices. (1) Smooth the distributions in some way, for instance with a Bayesian prior, or (similarly) taking the convex combination of the observation with some valid (nonzero) distribution. (The standard Bayesian approach is to use a Dirichlet prior, which amounts to treating each entry as a fraction $n_i / m$ where $m=\sum_i n_i$, and $n_i$ should be integer (but with your provided data this may get messy), and replacing these fractions with for instance $(n_i + 1) / (m+|x|)$ where $|x|$ is the number of atoms in the discrete distribution; the "convex combination" smoothing choice is similar, if $x$ is your observation and $\alpha \in (0,1]$, return $\alpha U_{|x|} + (1-\alpha)x$ where $U_{|X|}$ is the uniform distribution on $|x|$ points). points.) (2) Employ heuristics throwing out all the values that do not make sense, as suggested above. While I acknowledge that (2) is a 'convention', it is not justified. (People usually forgive convention, it by talking about limitsdoesn't really fit with the nature of these distributions, but as I will explain next, it doesn't make sense.)momentarily.

The longer answer is the mathematical reason why KL divergence can't handle these zeros, which requires information about Exponential family distributions (multinomial, gaussian, etc). Every exponential family distribution is defined relative to some base measure, and it must be nonzero everywhere on that base measure: this is true with multinomials, it is true with gaussians (covariance matrix must be full rank), etc. This arises because these distributions are the solution to an optimization problem which breaks down in the presence of those zeros. Anyway, so what needs to happen is that the relative base measure is the "tightest possible": in the case of multinomials, it is the uniform distribution on the nonzero entries, and in the Gaussian case, it is Lebesgue measure restricted to the affine subspace corresponding to the eigenspace of the provided covariance, shifted by the provided mean. The KL divergence (written as an integral) only makes sense if both distributions are relative to the same "tightest fit" measure.

To summarize, the invalidity of the formula in the presence of zeros isn't just some unfortunate hack, it is a deep issue intimately tied to how these distributions behave. The smoothing/Bayesian solution is thus better motivated: it nudges the distributions into validity. But many people simply choose to throw out those values (i.e., by erasing $0\ln(0)$ or $a\ln(a/0)$ and writing $0$ in its place).

1

I'll give a short answer and a long answer.

The short answer is that the KL divergence on multinomials is defined when they have only nonzero entries. When there are zero entries, you have two choices. (1) Smooth the distributions in some way, for instance with a Bayesian prior, or (similarly) taking the convex combination of the observation with some valid (nonzero) distribution (for instance, if $x$ is your observation and $\alpha \in (0,1]$, return $\alpha U_{|x|} + (1-\alpha)x$ where $U_{|X|}$ is the uniform distribution on $|x|$ points). (2) Employ heuristics throwing out all the values that do not make sense, as suggested above. While I acknowledge that (2) is a 'convention', it is not justified. (People usually forgive it by talking about limits, but as I will explain next, it doesn't make sense.)

The longer answer is the mathematical reason why KL divergence can't handle these zeros, which requires information about Exponential family distributions (multinomial, gaussian, etc). Every exponential family distribution is defined relative to some base measure, and it must be nonzero everywhere on that base measure: this is true with multinomials, it is true with gaussians (covariance matrix must be full rank), etc. This arises because these distributions are the solution to an optimization problem which breaks down in the presence of those zeros. Anyway, so what needs to happen is that the relative base measure is the "tightest possible": in the case of multinomials, it is the uniform distribution on the nonzero entries, and in the Gaussian case, it is Lebesgue measure restricted to the affine subspace corresponding to the eigenspace of the provided covariance, shifted by the provided mean. The KL divergence (written as an integral) only makes sense if both distributions are relative to the same "tightest fit" measure.

To summarize, the invalidity of the formula in the presence of zeros isn't just some unfortunate hack, it is a deep issue intimately tied to how these distributions behave. The smoothing/Bayesian solution is thus better motivated: it nudges the distributions into validity. But many people simply choose to throw out those values (i.e., by erasing $0\ln(0)$ or $a\ln(a/0)$ and writing $0$ in its place).