5
$\begingroup$

Consider two probability measures $\mu_0$ and $\mu_1$ on $\mathbb{R}^n$, such that $\mu_0\ll \mu_1$. Then I can define a "distance" like quantitiy $$ \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) $$ Is this quantity already known?

For simplicity assume that both measures are absolute continuous with respect to the Lebesgue measure and we denote by $p_0$ and $p_1$ the densities. Hence the condition $\mu_0\ll \mu_1$ states that $\mathrm{supp}(p_0)\subseteq \mathrm{supp}(p_1)$. And the quantity above can be bounded below by the Kullback-Leibler divergence $K(p_0|p_1)$, just by using the linearization of the logarithm $\log x \leq x-1$. $$ K(p_0| p_1) = \int p_0 \log \frac{p_0}{p_1} dx \leq \int \left(\frac{p_0^2}{p_1}-p_0\right)dx = \int \frac{p_0^2}{p_1^2}d\mu_1 - 1 = \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) $$ I'm especially interested in conditions on the distributions for which this quantity becomes $+\infty$, are there some simple characterizations?

For instance, if one considers two Gaussians with equal mean and different variances, hence $\mu_0 = \mathcal{N}(0,1)$ and $\mu_1 = \mathcal{N}(0,\sigma^2)$, then $$ \mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \begin{cases} \frac{\sigma}{\sqrt{2 \sigma^2-1}} - 1 &, \sigma^2 > \frac{1}{2} \\ +\infty &,\sigma^2 \leq \frac12 . \end{cases} $$ and as also $\mu_1 \ll \mu_0$ $$ \mathrm{Var}_{\mu_0}\left(\frac{\mathrm{d}\mu_1}{\mathrm{d}\mu_0}\right) = \begin{cases} \frac{1}{\sigma\sqrt{2-\sigma^2}} - 1 &, \sigma^2 < 2 \\ +\infty &,\sigma^2 \geq 2 . \end{cases} $$

One can obtain similar results for parameter regimes where this quantity is bounded if one coniders exponentials with different parameters or power laws with different exponents.

Further, ff one compares distributions with different tails (power law <-> exponential, exponential <-> Gaussian) than one distance will be always finite and the other distance with $\mu_0$ interchanged with $\mu_1$ will be infinite.

Hence the examples motivate the following non exact and even wrong characterization (cf. comment of Didier):

  • If $|\mathrm{supp} \mu_1|<\infty$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$.
  • If the tail of $\mu_1$ is lighter than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) = \infty$
  • If the tail of $\mu_1$ is heavier than the tail of $\mu_0$, then $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$
  • If the tails are equal strong, then one have to consider finer properties of the distributions.

Rephrased question: Is there a better characterization of $\mathrm{Var}_{\mu_1}\left(\frac{\mathrm{d}\mu_0}{\mathrm{d}\mu_1}\right) < \infty$?

$\endgroup$
5
  • $\begingroup$ @André What kind of condition, other than the ratio of densities $p_0^2/p_1$ being non integrable, would you like? $\endgroup$
    – Did
    Mar 13, 2011 at 17:58
  • $\begingroup$ @André By the way, your formula for Gaussians corresponds to $\mu_0$ and $\mu_1$ exchanged, that is, to the variance with respect to $\mu_0$ of the density $\mathrm{d}\mu_1/\mathrm{d}\mu_0$. $\endgroup$
    – Did
    Mar 14, 2011 at 6:58
  • $\begingroup$ @Didier: I tried to make the question more precise. But maybe you are right, that the best one can get is to consider the tails and to make a statement on this. Thank you for the further examples and I also considered some more, which now makes the quantity a little bit more clear to me. $\endgroup$ Mar 14, 2011 at 8:48
  • $\begingroup$ @André The first bulleted assertion in the edited part of your question is false (choose $\mu_0$ uniform on $(0,1)$ and $\mu_1$ with density $2x$ for $x$ in $(0,1)$). And I am not sure of what it is you call tail in the other bulleted assertions since the non integrability of $p_0^2/p_1$ can be caused by any region where $p_1$ is not lower bounded. $\endgroup$
    – Did
    Mar 14, 2011 at 18:16
  • $\begingroup$ @Didier: My characterization was rather stupid and only motivated by the examples I considered. $\endgroup$ Mar 14, 2011 at 19:43

2 Answers 2

6
$\begingroup$

The Kullback-Leibler divergence is a special case of Rényi divergence. In your notation, for $\alpha > 0$, the Rényi divergence of order $\alpha$ is defined by $$ D_\alpha(p_0,p_1) = \frac{1}{\alpha - 1} \log \left( \int \left(\frac{d\mu_0}{d\mu_1}\right)^\alpha d\mu_1 \right) = \frac{1}{\alpha - 1} \log \left(\int p_1 \frac{p_0^\alpha}{p_1^\alpha} dx\right) $$ when $\alpha \neq 1$, and $D_1$ is the Kullback-Leibler divergence. Thus the quantity you're interested is essentially the Rényi divergence of order 2: $$ \mathrm{Var}_{\mu_1}\left(\frac{d\mu_0}{d\mu_1}\right) = \exp D_2(p_0,p_1) - 1. $$

$\endgroup$
6
  • 1
    $\begingroup$ Thank you. After digging a little bit more on the internet I found the following nice overview on probability metrics: <www.math.hmc.edu/~su/papers.dir/metrics.pdf> The distance itself is actual called $\chi^2$ distance. On page 3 one finds the relationships between the different metrics, where one finds that the $\chi^2$ distance is one of the weakest distances. $\endgroup$ Mar 14, 2011 at 20:23
  • $\begingroup$ Thanks for the nice reference. I did once know and should have remembered that this is the $\chi^2$ distance, but I've been thinking about Rényi entropies lately so that distracted me. $\endgroup$ Mar 14, 2011 at 21:07
  • $\begingroup$ @André: incidentally, according to the way people usually use the words "weak" and "strong" in this context, the $\chi^2$ diestance is actually one of the strongest distances. That is, convergence in $\chi^2$ distance is a very strong property. $\endgroup$ Mar 15, 2011 at 13:28
  • $\begingroup$ @Mark: of course, you are right. I got confused through my application of this distance, which not relates to convergence. There this distance occured as constants in some inequalities, for which a strong norm means large constants. This lead me to the notion of weak in terms of this constants. $\endgroup$ Mar 25, 2011 at 17:46
  • 1
    $\begingroup$ @AndréSchlichting Unfortunately, the link you provided is down. Were you referring to Alison L. Gibbs and Francis Edward Su: On choosing and bounding probability metrics.? $\endgroup$ May 22, 2023 at 18:48
2
$\begingroup$

Surely you know this but as soon as there exists an event $A$ with positive $\mu_0(A)$ such that $\mu_1(A)\to0$ then your "distance" goes to infinity.

Consider for example $\mu_0$ uniform on $(0,1)$ and $\mu_1$ uniform on $(0,a)$ for a positive $a$. For every $a\ge1$, $\mu_0\ll\mu_1$ but $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)=a-1\to+\infty$ when $a\to+\infty$.

Edit A fixed-support example similar to the centered Gaussian one is $\mu_0$ exponential with parameter $1$ and $\mu_1$ exponential with positive parameter $a$. Then $\mbox{Var}_{\mu_1}(\mathrm{d}\mu_0/\mathrm{d}\mu_1)$ is $(1-a)^2/[1-(1-a)^2]$ for every $a<2$ and $+\infty$ for every $a\ge2$.

$\endgroup$
1
  • $\begingroup$ Right, this explains the blow up for $\sigma \to 0$ in the example of the two Gaussians, which is rather clear to me. More interesting is the blow up for $\sigma \to 2$. $\endgroup$ Mar 13, 2011 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.