Let $(\Omega, \mathcal{F}, \mu)$ be a probability space and let $Q$ be the uniform distribution on $(\Omega, \mathcal{F})$ such that $q = dQ / d\mu$ exists. Then the KL-divergence for some probability measure $P$ with $p = dP / \mu$ is given by $$ D_{\text{KL}}(P || Q) = \int_\Omega p \log \frac{p}{q} d\mu. $$i.e. we view the KL-divergence as a function of only $P$. If $\Omega$ is finite then I believe that the KL-divergence is bounded above by $|\Omega|\log|\Omega|$. But what if, more generally, $\Omega$ is compact and $\mathcal{F}$ is the Borel $\sigma$-algebra?
1 Answer
$\begingroup$
$\endgroup$
2
First, the definition of the KL divergence has nothing to do with compactness as it is defined entirely in terms of the density of one measure with respect to the other one.
Second, the KL divergence is not bounded even for finite spaces. Take $\Omega$ to be a two point set, $P$ be the distribution with the weights $(1/2,1/2)$, and $Q$ be $(\epsilon, 1-\epsilon)$ with $\epsilon$ close to 0.
-
$\begingroup$ Sorry, I meant if you fix $Q$ and view the KL-divergence just as a function of $P$. $\endgroup$– bvnCommented Feb 4, 2018 at 13:51
-
$\begingroup$ In the finite case, if you fix $Q$ and it has full support, then the derivatives $dP/dQ$ are bounded, so that $D(P||Q)$ is indeed bounded. On the other hand, for continuous measures nothing prevents you from making $dP/dQ$ (and therefore $D(P||Q)$) arbitrarily large (so that almost all mass of measure $P$ is concentrated on a set of a very small measure $Q$). $\endgroup$– R WCommented Feb 4, 2018 at 15:48