5
$\begingroup$

I am new here. I was reading another post here and this got me wondering what can be said about the size of the following kl divergence neighborhoods. Consider these two kl-divergence neighbourhood for a fixed distribution $P'$ and some $\alpha \geq 0$ $$ \mathbf{P} = \{P : D_{KL}(P||P') < \alpha\}\\ \mathbf{Q} = \{Q : D_{KL}(P'||Q) < \alpha\} $$

I am wondering anything can be said about $|\mathbf{P }|$ and $ |\mathbf{ Q}|$. Because the KL-divergence is asymmetric, I doubt they are not equal and the answer seems to be dependent on $P'$. When $\alpha=0$, the problem is trivial. I am curious about when $\alpha > 0$. I hope this question made sense. Thanks.

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Note that Pinsker's inequality bounds KL from below by the square of the total variation distance. This gives a bound which is similar for $\mathbf{P}$ and $\mathbf{Q}$. $\endgroup$
    – Dirk
    Commented Nov 7, 2013 at 8:39
  • 9
    $\begingroup$ What is meant by $|\mathbf P|$? The cardinality (this I assumed in my answer below), or the 'size' of a 'supremal' element in $\mathbf P$ (this would require a precise definition)? $\endgroup$
    – Joris Bierkens
    Commented Nov 7, 2013 at 8:48

1 Answer 1

Reset to default
1
$\begingroup$

Perhaps more a question for math.stackexchange.com.

Unless your probability space $(\Omega, \mathcal F)$ is trivial (i.e. $\mathcal F = \{ \emptyset, \Omega \})$, the sets $\mathbf P$ and $\mathbf Q$ will contain a continuum of probability distributions. (Consider probability measures of the form $\frac{d P}{dP'} =\exp(-X)/E [\exp(-X)]$ for random variables $X$. Not all random variables $X$ will work, but at least uncountably many).

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .