8
$\begingroup$

Are known expressions for total variation distance between $N(0,\sigma^2)$ and $N(0,\sigma^2+\epsilon)$ for small $\epsilon$? The only thing I seem to find is things are expression about the mean but not if we change variance slightly.

$\endgroup$
3
$\begingroup$

There's also a softer argument based on properties of the heat kernel, which applies in higher dimensions as well, in Lemma 4.9 of this paper of Klartag. It shows that in $n$ dimensions, the total variation distance between centered Gaussian distributions with covariances $\alpha I_n$ and $\beta I_n$ is at most $$ C \sqrt{n} \left|\frac{\beta}{\alpha} - 1\right|, $$ where $C > 0$ is an absolute constant (which can be made explicit if you want).

$\endgroup$
  • $\begingroup$ AH! interesting I am a probabilist but I am trying to show an integral satisfy a Holder continuous condition. Naturally I converted the language into total variation distance! You have just reverted it back to the original problem! $\endgroup$ – lost1 Feb 12 '14 at 14:00
  • 1
    $\begingroup$ This seems to easily follow from Pinsker's inequality. At least the \sqrt{n} part. The proof in the paper is very nice. $\endgroup$ – gondolier May 11 '16 at 6:59
  • $\begingroup$ @mr.gondolier If I did the calculation right, then Pinsker gives $\sqrt{\frac{n}{2}[ (\beta/\alpha)^2 - 1 - \log(\beta / \alpha)]}$, which reproduces this result when $\alpha$ and $\beta$ are very different, but which is worse when $\beta / \alpha \approx 1$. $\endgroup$ – Mark Meckes May 11 '16 at 15:01
  • $\begingroup$ @MarkMeckes I have $\sqrt{n/2 (\beta/\alpha-1-\log \beta/\alpha)}$. So asymptotically when $\beta/\alpha \approx 1$, KL divergence is quadratic and it gives the same result. $\endgroup$ – gondolier May 14 '16 at 5:42
  • $\begingroup$ @mr.gondolier You're right, I was alternately using $\alpha$ and $\beta$ as variance in one place and standard deviation in another. So Pinsker does give another nice approach, more elementar than the heat kernel approach, and simpler to carry out than the direct computation. $\endgroup$ – Mark Meckes May 14 '16 at 11:03
9
$\begingroup$

As marcoromito wrote, this is an elementary calculation. However, I thought I would record a nice approximation that I stumbled across. Whether it is new, I have no idea.

ADDED: The following sentence has changed.

By scaling, we can restrict ourselves to $\mathrm{tvd}(\epsilon)$ which is the total variation distance between $N(0,1)$ and $N(0,(1+\epsilon)^2)$ for $\epsilon\ge 0$. The points where the two densities are equal are $\pm x_0$ where $$x_0 = \frac{(1+\epsilon)\sqrt{2\ln(1+\epsilon)}}{\sqrt{\epsilon(2+\epsilon)}}.$$ Integrating the difference between the densities over $[-x_0,x_0]$ we find $$\mathrm{tvd}(\epsilon) = \mathrm{erf}\left( \frac{(1+\epsilon)\sqrt{\ln(1+\epsilon)}}{\sqrt{\epsilon(2+\epsilon)}}\right) - \mathrm{erf}\left( \frac{\sqrt{\ln(1+\epsilon)}}{\sqrt{\epsilon(2+\epsilon)}}\right). $$ The interesting thing is that for small $\epsilon\ge 0$ $$ \mathrm{tvd}(\epsilon) = \mathrm{tvd'}(\epsilon) + O(\epsilon^5)$$ where $$ \mathrm{tvd'}(\epsilon) = \frac{2^{3/2}\,\epsilon }{\pi^{1/2}e^{1/2}(2+\epsilon)}.$$ This rational approximation is remarkably precise for small $\epsilon$.

Experimentally (based on plotting the ratio in Maple and seeing a smooth curve), $$1 \le \frac{\mathrm{tvd}(\epsilon)}{\mathrm{tvd'}(\epsilon)} \le \sqrt{e\pi/8} \approx 1.03318$$ for all $\epsilon\gt 0$.

$\endgroup$
  • $\begingroup$ yeah, the last bit is exactly what i wanted. I got something similar, but not the same, so i need to chck. I was doing some pde stuff and estimating an integral against greens function of the heat equation, and want to show this integral is Holder and the expression basically the total variation distance of two gaussians. $\endgroup$ – lost1 Feb 8 '14 at 3:46
  • $\begingroup$ I am interested in how you got the 1.0332 and what is the bound for non unit variance? $\endgroup$ – lost1 Feb 8 '14 at 3:48
  • $\begingroup$ You have an extra factor of $2+\epsilon$ on the bottom, which I disagree with. $\endgroup$ – lost1 Feb 11 '14 at 15:11
  • $\begingroup$ @lost1: Sorry, I had a mistake. My formula is for standard deviation $1+\epsilon$, not variance $1+\epsilon$. I changed it. Do you agree now? $\endgroup$ – Brendan McKay Feb 12 '14 at 5:35
  • $\begingroup$ i verified this. working with variance $v$ and $v+\epsilon$ is (imo) nicer, because I needed to perturb it for a general $v$ but thank you. $\endgroup$ – lost1 Feb 12 '14 at 14:48
3
$\begingroup$

For two measures with densities f,g, the total variation distance is $$ \int_{f>g}(f(x)-g(x))\,dx $$ For two Gaussian measures with the same mean and different variances it is easy to identify the set $\{f>g\}$ and to obtain the formula for the distance in terms of the partition function of the standard Gaussian measure

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.