If you consider two probability distributions $p$ and $q$, one way to measure the distance between the two is the Kullback-Leibler divergence:

$$KL(p,q)=\int p \log (p/q) = E_p(\log p/q)$$

and this has many good properties.

I'm currently writing an article in which I want to use what I call the KL variance:

$$KL_{var}(p,q) = var_p(\log p/q) = \int p \log^2 (p/q) - KL(p,q)^2$$

Which also has many good properties

I have searched around quite a bit for references to this divergence, and I haven't found anything. Does anybody have an existing reference to this divergence ? Are there any names which would be slightly more catchy than KL-variance?

  • 2
    $\begingroup$ I haven't seen exactly this quantity considered, but something related is studied here (see in particular the discussion after Theorem 1.1): projecteuclid.org/euclid.aop/1312555807 $\endgroup$ Jun 30 '15 at 13:32
  • 2
    $\begingroup$ An aside: KL-divergence also has a (better) name: "relative entropy". $\endgroup$ Jun 30 '15 at 17:03
  • 1
    $\begingroup$ Relative entropy is a pretty poor name, to be honest. It implies that KL(p,q) is somehow related to the difference of the entropies H(p) - H(q) or something like that, which it isn't at all. It's useful to keep in mind the fact the KL(p,q) represents the expected number of extra bits, but that name is confusing imo $\endgroup$ Jun 30 '15 at 20:46

Too late to help you I guess but I'll leave this here for future reference. This quantity shows up a lot in Bayesian nonparametrics, when proving (frequentist) posterior contraction rates, so "posterior contraction rates" is a useful search. I don't know of any snappy names for it, but theorem 8.3 (and the comment afterwards) in Ghosal, Ghosh, van der Vaart, Convergence rates of posterior distributions, 2000, gives the following bound:

$$var_p \log(p/q) \leq 4h^2(p,q) \lVert{p/q}\rVert_\infty,$$ where $h$ is the Hellinger distance $h(p,q)^2=\int (\sqrt{p}-\sqrt{q})^2$:



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.