# Size of KL-divergence neighbourhoods

I am new here. I was reading another post here and this got me wondering what can be said about the size of the following kl divergence neighborhoods. Consider these two kl-divergence neighbourhood for a fixed distribution $P'$ and some $\alpha \geq 0$ $$\mathbf{P} = \{P : D_{KL}(P||P') < \alpha\}\\ \mathbf{Q} = \{Q : D_{KL}(P'||Q) < \alpha\}$$

I am wondering anything can be said about $|\mathbf{P }|$ and $|\mathbf{ Q}|$. Because the KL-divergence is asymmetric, I doubt they are not equal and the answer seems to be dependent on $P'$. When $\alpha=0$, the problem is trivial. I am curious about when $\alpha > 0$. I hope this question made sense. Thanks.

• Note that Pinsker's inequality bounds KL from below by the square of the total variation distance. This gives a bound which is similar for $\mathbf{P}$ and $\mathbf{Q}$.
– Dirk
Commented Nov 7, 2013 at 8:39
• What is meant by $|\mathbf P|$? The cardinality (this I assumed in my answer below), or the 'size' of a 'supremal' element in $\mathbf P$ (this would require a precise definition)? Commented Nov 7, 2013 at 8:48

Unless your probability space $(\Omega, \mathcal F)$ is trivial (i.e. $\mathcal F = \{ \emptyset, \Omega \})$, the sets $\mathbf P$ and $\mathbf Q$ will contain a continuum of probability distributions. (Consider probability measures of the form $\frac{d P}{dP'} =\exp(-X)/E [\exp(-X)]$ for random variables $X$. Not all random variables $X$ will work, but at least uncountably many).