0
$\begingroup$

Let $(\Omega, \mathcal{F}, \mu)$ be a probability space and let $Q$ be the uniform distribution on $(\Omega, \mathcal{F})$ such that $q = dQ / d\mu$ exists. Then the KL-divergence for some probability measure $P$ with $p = dP / \mu$ is given by $$ D_{\text{KL}}(P || Q) = \int_\Omega p \log \frac{p}{q} d\mu. $$i.e. we view the KL-divergence as a function of only $P$. If $\Omega$ is finite then I believe that the KL-divergence is bounded above by $|\Omega|\log|\Omega|$. But what if, more generally, $\Omega$ is compact and $\mathcal{F}$ is the Borel $\sigma$-algebra?

$\endgroup$

1 Answer 1

0
$\begingroup$

First, the definition of the KL divergence has nothing to do with compactness as it is defined entirely in terms of the density of one measure with respect to the other one.

Second, the KL divergence is not bounded even for finite spaces. Take $\Omega$ to be a two point set, $P$ be the distribution with the weights $(1/2,1/2)$, and $Q$ be $(\epsilon, 1-\epsilon)$ with $\epsilon$ close to 0.

$\endgroup$
2
  • $\begingroup$ Sorry, I meant if you fix $Q$ and view the KL-divergence just as a function of $P$. $\endgroup$
    – bvn
    Feb 4, 2018 at 13:51
  • $\begingroup$ In the finite case, if you fix $Q$ and it has full support, then the derivatives $dP/dQ$ are bounded, so that $D(P||Q)$ is indeed bounded. On the other hand, for continuous measures nothing prevents you from making $dP/dQ$ (and therefore $D(P||Q)$) arbitrarily large (so that almost all mass of measure $P$ is concentrated on a set of a very small measure $Q$). $\endgroup$
    – R W
    Feb 4, 2018 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.