distance in terms of the variance between two absolutely continuous probability measures - MathOverflow most recent 30 from http://mathoverflow.net 2013-06-20T04:53:36Z http://mathoverflow.net/feeds/question/58321 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/58321/distance-in-terms-of-the-variance-between-two-absolutely-continuous-probability-m distance in terms of the variance between two absolutely continuous probability measures André Schlichting 2011-03-13T10:18:31Z 2011-03-14T20:15:42Z <p>Consider two probability measures $\mu_0$ and $\mu_1$ on $\mathbb{R}^n$, such that $\mu_0\ll \mu_1$. Then I can define a "distance" like quantitiy $$\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right)$$ Is this quantity already known?</p> <p>For simplicity assume that both measures are absolute continuous with respect to the Lebesgue measure and we denote by $p_0$ and $p_1$ the densities. Hence the condition $\mu_0\ll \mu_1$ states that $\mathrm{supp}(p_0)\subseteq \mathrm{supp}(p_1)$. And the quantity above can be bounded below by the Kullback-Leibler divergence $K(p_0|p_1)$, just by using the linearization of the logarithm $\log x \leq x-1$. $$K(p_0| p_1) = \int p_0 \log \frac{p_0}{p_1} dx \leq \int \left(\frac{p_0^2}{p_1}-p_0\right)dx = \int \frac{p_0^2}{p_1^2}d\mu_1 - 1 = \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right)$$ I'm especially interested in conditions on the distributions for which this quantity becomes $+\infty$, are there some simple characterizations?</p> <p>For instance, if one considers two Gaussians with equal mean and different variances, hence $\mu_0 = \mathcal{N}(0,1)$ and $\mu_1 = \mathcal{N}(0,\sigma^2)$, then <code>$$\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \begin{cases} \frac{\sigma}{\sqrt{2 \sigma^2-1}} - 1 &amp;, \sigma^2 &gt; \frac{1}{2} \\ +\infty &amp;,\sigma^2 \leq \frac12 . \end{cases}$$</code> and as also $\mu_1 \ll \mu_0$ <code>$$\mathrm{Var}_{\mu_0}\left(\frac{\mathrm{d}\mu_1}{\mathrm{d}\mu_0}\right) = \begin{cases} \frac{1}{\sigma\sqrt{2-\sigma^2}} - 1 &amp;, \sigma^2 &lt; 2 \\ +\infty &amp;,\sigma^2 \geq 2 . \end{cases}$$</code></p> <p>One can obtain similar results for parameter regimes where this quantity is bounded if one coniders exponentials with different parameters or power laws with different exponents. </p> <p>Further, ff one compares distributions with different tails (power law &lt;-> exponential, exponential &lt;-> Gaussian) than one distance will be always finite and the other distance with $\mu_0$ interchanged with $\mu_1$ will be infinite.</p> <p>Hence the examples motivate the following non exact and even wrong characterization (cf. comment of Didier):</p> <ul> <li>If $|\mathrm{supp} \mu_1|&lt;\infty$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) &lt; \infty$.</li> <li>If the tail of $\mu_1$ is lighter than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \infty$</li> <li>If the tail of $\mu_1$ is heavier than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) &lt; \infty$</li> <li>If the tails are equal strong, then one have to consider finer properties of the distributions.</li> </ul> <p><strong>Rephrased question:</strong> Is there a better characterization of $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) &lt; \infty$?</p> http://mathoverflow.net/questions/58321/distance-in-terms-of-the-variance-between-two-absolutely-continuous-probability-m/58340#58340 Answer by Didier Piau for distance in terms of the variance between two absolutely continuous probability measures Didier Piau 2011-03-13T15:06:10Z 2011-03-13T17:20:13Z <p>Surely you know this but as soon as there exists an event $A$ with positive $\mu_0(A)$ such that $\mu_1(A)\to0$ then your "distance" goes to infinity.</p> <p>Consider for example $\mu_0$ uniform on $(0,1)$ and $\mu_1$ uniform on $(0,a)$ for a positive $a$. For every $a\ge1$, $\mu_0\ll\mu_1$ but $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)=a-1\to+\infty$ when $a\to+\infty$.</p> <p><strong>Edit</strong> A fixed-support example similar to the centered Gaussian one is $\mu_0$ exponential with parameter $1$ and $\mu_1$ exponential with positive parameter $a$. Then $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)$ is $(1-a)^2/[1-(1-a)^2]$ for every $a&lt;2$ and $+\infty$ for every $a\ge2$.</p> http://mathoverflow.net/questions/58321/distance-in-terms-of-the-variance-between-two-absolutely-continuous-probability-m/58442#58442 Answer by Mark Meckes for distance in terms of the variance between two absolutely continuous probability measures Mark Meckes 2011-03-14T16:18:31Z 2011-03-14T16:18:31Z <p>The Kullback-Leibler divergence is a special case of <a href="http://en.wikipedia.org/wiki/R%25C3%25A9nyi_entropy#R.C3.A9nyi_divergence" rel="nofollow">Rényi divergence</a>. In your notation, for $\alpha > 0$, the Rényi divergence of order $\alpha$ is defined by $$D_\alpha(p_0,p_1) = \frac{1}{\alpha - 1} \log \left( \int \left(\frac{d\mu_0}{d\mu_1}\right)^\alpha d\mu_1 \right) = \frac{1}{\alpha - 1} \log \left(\int p_1 \frac{p_0^\alpha}{p_1^\alpha} dx\right)$$ when $\alpha \neq 1$, and $D_1$ is the Kullback-Leibler divergence. Thus the quantity you're interested is essentially the Rényi divergence of order 2: $$\mathrm{Var}_{\mu_1}\left(\frac{d\mu_0}{d\mu_1}\right) = \exp D_2(p_0,p_1) - 1.$$</p>