distance in terms of the variance between two absolutely continuous probability measures - MathOverflow

distance in terms of the variance between two absolutely continuous probability measures

2011-03-13T10:18:31Z

Consider two probability measures $\mu_0$ and $\mu_1$ on $\mathbb{R}^n$, such that $\mu_0\ll \mu_1$. Then I can define a "distance" like quantitiy $$ \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) $$ Is this quantity already known?

For simplicity assume that both measures are absolute continuous with respect to the Lebesgue measure and we denote by $p_0$ and $p_1$ the densities. Hence the condition $\mu_0\ll \mu_1$ states that $\mathrm{supp}(p_0)\subseteq \mathrm{supp}(p_1)$. And the quantity above can be bounded below by the Kullback-Leibler divergence $K(p_0|p_1)$, just by using the linearization of the logarithm $\log x \leq x-1$. $$ K(p_0| p_1) = \int p_0 \log \frac{p_0}{p_1} dx \leq \int \left(\frac{p_0^2}{p_1}-p_0\right)dx = \int \frac{p_0^2}{p_1^2}d\mu_1 - 1 = \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) $$ I'm especially interested in conditions on the distributions for which this quantity becomes $+\infty$, are there some simple characterizations?

For instance, if one considers two Gaussians with equal mean and different variances, hence $\mu_0 = \mathcal{N}(0,1)$ and $\mu_1 = \mathcal{N}(0,\sigma^2)$, then $$ \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \begin{cases} \frac{\sigma}{\sqrt{2 \sigma^2-1}} - 1 &, \sigma^2 > \frac{1}{2} \\ +\infty &,\sigma^2 \leq \frac12 . \end{cases} $$ and as also $\mu_1 \ll \mu_0$ $$ \mathrm{Var}_{\mu_0}\left(\frac{\mathrm{d}\mu_1}{\mathrm{d}\mu_0}\right) = \begin{cases} \frac{1}{\sigma\sqrt{2-\sigma^2}} - 1 &, \sigma^2 < 2 \\ +\infty &,\sigma^2 \geq 2 . \end{cases} $$

One can obtain similar results for parameter regimes where this quantity is bounded if one coniders exponentials with different parameters or power laws with different exponents.

Further, ff one compares distributions with different tails (power law <-> exponential, exponential <-> Gaussian) than one distance will be always finite and the other distance with $\mu_0$ interchanged with $\mu_1$ will be infinite.

Hence the examples motivate the following non exact and even wrong characterization (cf. comment of Didier):

If $|\mathrm{supp} \mu_1|<\infty$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$.
If the tail of $\mu_1$ is lighter than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \infty$
If the tail of $\mu_1$ is heavier than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$
If the tails are equal strong, then one have to consider finer properties of the distributions.

Rephrased question: Is there a better characterization of $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$?

Answer by Didier Piau for distance in terms of the variance between two absolutely continuous probability measures

2011-03-13T15:06:10Z

Surely you know this but as soon as there exists an event $A$ with positive $\mu_0(A)$ such that $\mu_1(A)\to0$ then your "distance" goes to infinity.

Consider for example $\mu_0$ uniform on $(0,1)$ and $\mu_1$ uniform on $(0,a)$ for a positive $a$. For every $a\ge1$, $\mu_0\ll\mu_1$ but $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)=a-1\to+\infty$ when $a\to+\infty$.

Edit A fixed-support example similar to the centered Gaussian one is $\mu_0$ exponential with parameter $1$ and $\mu_1$ exponential with positive parameter $a$. Then $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)$ is $(1-a)^2/[1-(1-a)^2]$ for every $a<2$ and $+\infty$ for every $a\ge2$.

Answer by Mark Meckes for distance in terms of the variance between two absolutely continuous probability measures

2011-03-14T16:18:31Z

The Kullback-Leibler divergence is a special case of Rényi divergence. In your notation, for $\alpha > 0$, the Rényi divergence of order $\alpha$ is defined by $$ D_\alpha(p_0,p_1) = \frac{1}{\alpha - 1} \log \left( \int \left(\frac{d\mu_0}{d\mu_1}\right)^\alpha d\mu_1 \right) = \frac{1}{\alpha - 1} \log \left(\int p_1 \frac{p_0^\alpha}{p_1^\alpha} dx\right) $$ when $\alpha \neq 1$, and $D_1$ is the Kullback-Leibler divergence. Thus the quantity you're interested is essentially the Rényi divergence of order 2: $$ \mathrm{Var}_{\mu_1}\left(\frac{d\mu_0}{d\mu_1}\right) = \exp D_2(p_0,p_1) - 1. $$