# Kullback Leibler “variance”: does that divergence have a name?

If you consider two probability distributions $p$ and $q$, one way to measure the distance between the two is the Kullback-Leibler divergence:

$$KL(p,q)=\int p \log (p/q) = E_p(\log p/q)$$

and this has many good properties.

I'm currently writing an article in which I want to use what I call the KL variance:

$$KL_{var}(p,q) = var_p(\log p/q) = \int p \log^2 (p/q) - KL(p,q)^2$$

Which also has many good properties

I have searched around quite a bit for references to this divergence, and I haven't found anything. Does anybody have an existing reference to this divergence ? Are there any names which would be slightly more catchy than KL-variance?

• I haven't seen exactly this quantity considered, but something related is studied here (see in particular the discussion after Theorem 1.1): projecteuclid.org/euclid.aop/1312555807 – Mark Meckes Jun 30 '15 at 13:32
• An aside: KL-divergence also has a (better) name: "relative entropy". – Nicolas Schmidt Jun 30 '15 at 17:03
• Relative entropy is a pretty poor name, to be honest. It implies that KL(p,q) is somehow related to the difference of the entropies H(p) - H(q) or something like that, which it isn't at all. It's useful to keep in mind the fact the KL(p,q) represents the expected number of extra bits, but that name is confusing imo – Guillaume Dehaene Jun 30 '15 at 20:46

$$var_p \log(p/q) \leq 4h^2(p,q) \lVert{p/q}\rVert_\infty,$$ where $h$ is the Hellinger distance $h(p,q)^2=\int (\sqrt{p}-\sqrt{q})^2$: