6
$\begingroup$

If you consider two probability distributions $p$ and $q$, one way to measure the distance between the two is the Kullback-Leibler divergence:

$$KL(p,q)=\int p \log (p/q) = E_p(\log p/q)$$

and this has many good properties.

I'm currently writing an article in which I want to use what I call the KL variance:

$$KL_{var}(p,q) = var_p(\log p/q) = \int p \log^2 (p/q) - KL(p,q)^2$$

Which also has many good properties

I have searched around quite a bit for references to this divergence, and I haven't found anything. Does anybody have an existing reference to this divergence ? Are there any names which would be slightly more catchy than KL-variance?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ I haven't seen exactly this quantity considered, but something related is studied here (see in particular the discussion after Theorem 1.1): projecteuclid.org/euclid.aop/1312555807 $\endgroup$
    – Mark Meckes
    Commented Jun 30, 2015 at 13:32
  • 2
    $\begingroup$ An aside: KL-divergence also has a (better) name: "relative entropy". $\endgroup$
    – Nicolas Schmidt
    Commented Jun 30, 2015 at 17:03
  • 1
    $\begingroup$ Relative entropy is a pretty poor name, to be honest. It implies that KL(p,q) is somehow related to the difference of the entropies H(p) - H(q) or something like that, which it isn't at all. It's useful to keep in mind the fact the KL(p,q) represents the expected number of extra bits, but that name is confusing imo $\endgroup$
    – Guillaume Dehaene
    Commented Jun 30, 2015 at 20:46

1 Answer 1

Reset to default
6
$\begingroup$

Too late to help you I guess but I'll leave this here for future reference. This quantity shows up a lot in Bayesian nonparametrics, when proving (frequentist) posterior contraction rates, so "posterior contraction rates" is a useful search. I don't know of any snappy names for it, but theorem 8.3 (and the comment afterwards) in Ghosal, Ghosh, van der Vaart, Convergence rates of posterior distributions, 2000, gives the following bound:

$$var_p \log(p/q) \leq 4h^2(p,q) \lVert{p/q}\rVert_\infty,$$ where $h$ is the Hellinger distance $h(p,q)^2=\int (\sqrt{p}-\sqrt{q})^2$:

https://projecteuclid.org/euclid.aos/1016218228

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .