9
$\begingroup$

Are known expressions for total variation distance between $N(0,\sigma^2)$ and $N(0,\sigma^2+\epsilon)$ for small $\epsilon$? The only thing I seem to find is things are expression about the mean but not if we change variance slightly.

$\endgroup$

3 Answers 3

4
$\begingroup$

There's also a softer argument based on properties of the heat kernel, which applies in higher dimensions as well, in Lemma 4.9 of this paper of Klartag. It shows that in $n$ dimensions, the total variation distance between centered Gaussian distributions with covariances $\alpha I_n$ and $\beta I_n$ is at most $$ C \sqrt{n} \left|\frac{\beta}{\alpha} - 1\right|, $$ where $C > 0$ is an absolute constant (which can be made explicit if you want).

$\endgroup$
5
  • $\begingroup$ AH! interesting I am a probabilist but I am trying to show an integral satisfy a Holder continuous condition. Naturally I converted the language into total variation distance! You have just reverted it back to the original problem! $\endgroup$
    – lost1
    Feb 12, 2014 at 14:00
  • 1
    $\begingroup$ This seems to easily follow from Pinsker's inequality. At least the \sqrt{n} part. The proof in the paper is very nice. $\endgroup$
    – gondolier
    May 11, 2016 at 6:59
  • $\begingroup$ @mr.gondolier If I did the calculation right, then Pinsker gives $\sqrt{\frac{n}{2}[ (\beta/\alpha)^2 - 1 - \log(\beta / \alpha)]}$, which reproduces this result when $\alpha$ and $\beta$ are very different, but which is worse when $\beta / \alpha \approx 1$. $\endgroup$ May 11, 2016 at 15:01
  • $\begingroup$ @MarkMeckes I have $\sqrt{n/2 (\beta/\alpha-1-\log \beta/\alpha)}$. So asymptotically when $\beta/\alpha \approx 1$, KL divergence is quadratic and it gives the same result. $\endgroup$
    – gondolier
    May 14, 2016 at 5:42
  • $\begingroup$ @mr.gondolier You're right, I was alternately using $\alpha$ and $\beta$ as variance in one place and standard deviation in another. So Pinsker does give another nice approach, more elementar than the heat kernel approach, and simpler to carry out than the direct computation. $\endgroup$ May 14, 2016 at 11:03
10
$\begingroup$

As marcoromito wrote, this is an elementary calculation. However, I thought I would record a nice approximation that I stumbled across. Whether it is new, I have no idea.

ADDED: The following sentence has changed.

By scaling, we can restrict ourselves to $\mathrm{tvd}(\epsilon)$ which is the total variation distance between $N(0,1)$ and $N(0,(1+\epsilon)^2)$ for $\epsilon\ge 0$. The points where the two densities are equal are $\pm x_0$ where $$x_0 = \frac{(1+\epsilon)\sqrt{2\ln(1+\epsilon)}}{\sqrt{\epsilon(2+\epsilon)}}.$$ Integrating the difference between the densities over $[-x_0,x_0]$ we find $$\mathrm{tvd}(\epsilon) = \mathrm{erf}\left( \frac{(1+\epsilon)\sqrt{\ln(1+\epsilon)}}{\sqrt{\epsilon(2+\epsilon)}}\right) - \mathrm{erf}\left( \frac{\sqrt{\ln(1+\epsilon)}}{\sqrt{\epsilon(2+\epsilon)}}\right). $$ The interesting thing is that for small $\epsilon\ge 0$ $$ \mathrm{tvd}(\epsilon) = \mathrm{tvd'}(\epsilon) + O(\epsilon^5)$$ where $$ \mathrm{tvd'}(\epsilon) = \frac{2^{3/2}\,\epsilon }{\pi^{1/2}e^{1/2}(2+\epsilon)}.$$ This rational approximation is remarkably precise for small $\epsilon$.

Experimentally (based on plotting the ratio in Maple and seeing a smooth curve), $$1 \le \frac{\mathrm{tvd}(\epsilon)}{\mathrm{tvd'}(\epsilon)} \le \sqrt{e\pi/8} \approx 1.03318$$ for all $\epsilon\gt 0$.

$\endgroup$
5
  • $\begingroup$ yeah, the last bit is exactly what i wanted. I got something similar, but not the same, so i need to chck. I was doing some pde stuff and estimating an integral against greens function of the heat equation, and want to show this integral is Holder and the expression basically the total variation distance of two gaussians. $\endgroup$
    – lost1
    Feb 8, 2014 at 3:46
  • $\begingroup$ I am interested in how you got the 1.0332 and what is the bound for non unit variance? $\endgroup$
    – lost1
    Feb 8, 2014 at 3:48
  • $\begingroup$ You have an extra factor of $2+\epsilon$ on the bottom, which I disagree with. $\endgroup$
    – lost1
    Feb 11, 2014 at 15:11
  • $\begingroup$ @lost1: Sorry, I had a mistake. My formula is for standard deviation $1+\epsilon$, not variance $1+\epsilon$. I changed it. Do you agree now? $\endgroup$ Feb 12, 2014 at 5:35
  • $\begingroup$ i verified this. working with variance $v$ and $v+\epsilon$ is (imo) nicer, because I needed to perturb it for a general $v$ but thank you. $\endgroup$
    – lost1
    Feb 12, 2014 at 14:48
3
$\begingroup$

For two measures with densities f,g, the total variation distance is $$ \int_{f>g}(f(x)-g(x))\,dx $$ For two Gaussian measures with the same mean and different variances it is easy to identify the set $\{f>g\}$ and to obtain the formula for the distance in terms of the partition function of the standard Gaussian measure

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.