I am new here. I was reading another post here and this got me wondering what can be said about the size of the following kl divergence neighborhoods. Consider these two kl-divergence neighbourhood for a fixed distribution $P'$ and some $\alpha \geq 0$ $$ \mathbf{P} = \{P : D_{KL}(P||P') < \alpha\}\\ \mathbf{Q} = \{Q : D_{KL}(P'||Q) < \alpha\} $$

I am wondering anything can be said about $|\mathbf{P }|$ and $ |\mathbf{ Q}|$. Because the KL-divergence is asymmetric, I doubt they are not equal and the answer seems to be dependent on $P'$. When $\alpha=0$, the problem is trivial. I am curious about when $\alpha > 0$. I hope this question made sense. Thanks.

  • 3
    $\begingroup$ Note that Pinsker's inequality bounds KL from below by the square of the total variation distance. This gives a bound which is similar for $\mathbf{P}$ and $\mathbf{Q}$. $\endgroup$
    – Dirk
    Commented Nov 7, 2013 at 8:39
  • 9
    $\begingroup$ What is meant by $|\mathbf P|$? The cardinality (this I assumed in my answer below), or the 'size' of a 'supremal' element in $\mathbf P$ (this would require a precise definition)? $\endgroup$ Commented Nov 7, 2013 at 8:48

1 Answer 1


Perhaps more a question for math.stackexchange.com.

Unless your probability space $(\Omega, \mathcal F)$ is trivial (i.e. $\mathcal F = \{ \emptyset, \Omega \})$, the sets $\mathbf P$ and $\mathbf Q$ will contain a continuum of probability distributions. (Consider probability measures of the form $\frac{d P}{dP'} =\exp(-X)/E [\exp(-X)]$ for random variables $X$. Not all random variables $X$ will work, but at least uncountably many).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.