Consider two probability measures $\mu_0$ and $\mu_1$ on $\mathbb{R}^n$, such that $\mu_0\ll \mu_1$. Then I can define a "distance" like quantitiy $$ \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) $$ Is this quantity already known?

For simplicity assume that both measures are absolute continuous with respect to the Lebesgue measure and we denote by $p_0$ and $p_1$ the densities. Hence the condition $\mu_0\ll \mu_1$ states that $\mathrm{supp}(p_0)\subseteq \mathrm{supp}(p_1)$. And the quantity above can be bounded below by the Kullback-Leibler divergence $K(p_0|p_1)$, just by using the linearization of the logarithm $\log x \leq x-1$. $$ K(p_0| p_1) = \int p_0 \log \frac{p_0}{p_1} dx \leq \int \left(\frac{p_0^2}{p_1}-p_0\right)dx = \int \frac{p_0^2}{p_1^2}d\mu_1 - 1 = \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) $$ I'm especially interested in conditions on the distributions for which this quantity becomes $+\infty$, are there some simple characterizations?

For instance, if one considers two Gaussians with equal mean and different variances, hence $\mu_0 = \mathcal{N}(0,1)$ and $\mu_1 = \mathcal{N}(0,\sigma^2)$, then $$ \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \begin{cases} \frac{\sigma}{\sqrt{2 \sigma^2-1}} - 1 &, \sigma^2 > \frac{1}{2} \\ +\infty &,\sigma^2 \leq \frac12 . \end{cases} $$ and as also $\mu_1 \ll \mu_0$ $$ \mathrm{Var}_{\mu_0}\left(\frac{\mathrm{d}\mu_1}{\mathrm{d}\mu_0}\right) = \begin{cases} \frac{1}{\sigma\sqrt{2-\sigma^2}} - 1 &, \sigma^2 < 2 \\ +\infty &,\sigma^2 \geq 2 . \end{cases} $$

One can obtain similar results for parameter regimes where this quantity is bounded if one coniders exponentials with different parameters or power laws with different exponents.

Further, ff one compares distributions with different tails (power law <-> exponential, exponential <-> Gaussian) than one distance will be always finite and the other distance with $\mu_0$ interchanged with $\mu_1$ will be infinite.

Hence the examples motivate the following non exact and even wrong characterization (cf. comment of Didier):

  • If $|\mathrm{supp} \mu_1|<\infty$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$.
  • If the tail of $\mu_1$ is lighter than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \infty$
  • If the tail of $\mu_1$ is heavier than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$
  • If the tails are equal strong, then one have to consider finer properties of the distributions.

Rephrased question: Is there a better characterization of $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$?

  • $\begingroup$ @André What kind of condition, other than the ratio of densities $p_0^2/p_1$ being non integrable, would you like? $\endgroup$ – Did Mar 13 '11 at 17:58
  • $\begingroup$ @André By the way, your formula for Gaussians corresponds to $\mu_0$ and $\mu_1$ exchanged, that is, to the variance with respect to $\mu_0$ of the density $\mathrm{d}\mu_1/\mathrm{d}\mu_0$. $\endgroup$ – Did Mar 14 '11 at 6:58
  • $\begingroup$ @Didier: I tried to make the question more precise. But maybe you are right, that the best one can get is to consider the tails and to make a statement on this. Thank you for the further examples and I also considered some more, which now makes the quantity a little bit more clear to me. $\endgroup$ – André Schlichting Mar 14 '11 at 8:48
  • $\begingroup$ @André The first bulleted assertion in the edited part of your question is false (choose $\mu_0$ uniform on $(0,1)$ and $\mu_1$ with density $2x$ for $x$ in $(0,1)$). And I am not sure of what it is you call tail in the other bulleted assertions since the non integrability of $p_0^2/p_1$ can be caused by any region where $p_1$ is not lower bounded. $\endgroup$ – Did Mar 14 '11 at 18:16
  • $\begingroup$ @Didier: My characterization was rather stupid and only motivated by the examples I considered. $\endgroup$ – André Schlichting Mar 14 '11 at 19:43

The Kullback-Leibler divergence is a special case of Rényi divergence. In your notation, for $\alpha > 0$, the Rényi divergence of order $\alpha$ is defined by $$ D_\alpha(p_0,p_1) = \frac{1}{\alpha - 1} \log \left( \int \left(\frac{d\mu_0}{d\mu_1}\right)^\alpha d\mu_1 \right) = \frac{1}{\alpha - 1} \log \left(\int p_1 \frac{p_0^\alpha}{p_1^\alpha} dx\right) $$ when $\alpha \neq 1$, and $D_1$ is the Kullback-Leibler divergence. Thus the quantity you're interested is essentially the Rényi divergence of order 2: $$ \mathrm{Var}_{\mu_1}\left(\frac{d\mu_0}{d\mu_1}\right) = \exp D_2(p_0,p_1) - 1. $$

  • 1
    $\begingroup$ Thank you. After digging a little bit more on the internet I found the following nice overview on probability metrics: <www.math.hmc.edu/~su/papers.dir/metrics.pdf> The distance itself is actual called $\chi^2$ distance. On page 3 one finds the relationships between the different metrics, where one finds that the $\chi^2$ distance is one of the weakest distances. $\endgroup$ – André Schlichting Mar 14 '11 at 20:23
  • $\begingroup$ Thanks for the nice reference. I did once know and should have remembered that this is the $\chi^2$ distance, but I've been thinking about Rényi entropies lately so that distracted me. $\endgroup$ – Mark Meckes Mar 14 '11 at 21:07
  • $\begingroup$ @André: incidentally, according to the way people usually use the words "weak" and "strong" in this context, the $\chi^2$ diestance is actually one of the strongest distances. That is, convergence in $\chi^2$ distance is a very strong property. $\endgroup$ – Mark Meckes Mar 15 '11 at 13:28
  • $\begingroup$ @Mark: of course, you are right. I got confused through my application of this distance, which not relates to convergence. There this distance occured as constants in some inequalities, for which a strong norm means large constants. This lead me to the notion of weak in terms of this constants. $\endgroup$ – André Schlichting Mar 25 '11 at 17:46

Surely you know this but as soon as there exists an event $A$ with positive $\mu_0(A)$ such that $\mu_1(A)\to0$ then your "distance" goes to infinity.

Consider for example $\mu_0$ uniform on $(0,1)$ and $\mu_1$ uniform on $(0,a)$ for a positive $a$. For every $a\ge1$, $\mu_0\ll\mu_1$ but $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)=a-1\to+\infty$ when $a\to+\infty$.

Edit A fixed-support example similar to the centered Gaussian one is $\mu_0$ exponential with parameter $1$ and $\mu_1$ exponential with positive parameter $a$. Then $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)$ is $(1-a)^2/[1-(1-a)^2]$ for every $a<2$ and $+\infty$ for every $a\ge2$.

  • $\begingroup$ Right, this explains the blow up for $\sigma \to 0$ in the example of the two Gaussians, which is rather clear to me. More interesting is the blow up for $\sigma \to 2$. $\endgroup$ – André Schlichting Mar 13 '11 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.