# Asymptotic bound on the total variation distance between a standard multivariate normal and a simple mixture

Let $$P = N(\vec{0}, I^d)$$ be a standard multivariate Gaussian distribution in $$d$$ dimensions. Let $$Q$$ be distributed the same as $$P$$, except that samples from $$Q$$ have one of their coordinates, chosen uniformly at random from $$1$$ to $$d$$, distributed as $$N(\mu, \sigma^2)$$ instead.

That is, $$Q$$ is a mixture of $$d$$ multivariate Gaussians, each with weight $$\frac1d$$. The $$i$$th mixture component has mean $$\vec{0} + \vec{e}_i \cdot \mu$$ and variance with diagonal $$\vec{1} + \vec{e}_i \cdot (\sigma^2 - 1)$$.

I'd like to bound the total variation distance between $$P$$ and $$Q$$, as a function of $$d$$ (when $$\mu$$ and $$\sigma$$ are constants that do not depend on $$d$$).

My intuition is that $$Q$$ is very close to a multivariate Gaussian $$N(\frac{\vec{\mu}}{d}, I^d \cdot (1+\frac{\sigma^2}{d}+ o(\frac1d))$$, which would give a Kullback–Leibler divergence of $$D_{KL}(P || Q) = O(\frac1d)$$ and thus a total variation distance of $$O(\frac{1}{\sqrt d})$$.

The case where $$\sigma^2=1$$ might be easier and sufficient for what I need. Any suggestions on how I could bound the TV distance between P and Q?

$$\newcommand{\m}{\vec{\mu}} \newcommand{\e}{\vec{e}} \newcommand{\x}{\vec{x}}$$ Let $$n:=d$$. Assume that $$\sigma=1$$ and $$\mu\ne0$$. Then for the densities $$\begin{equation*} f_Q(\x)=(2\pi)^{-n/2}\,\frac1n\,\sum_1^n e^{-|\x-\mu\e_i|^2/2} \end{equation*}$$ and $$\begin{equation*} f_P(\x)=(2\pi)^{-n/2}\,e^{-|\x|^2/2} \end{equation*}$$ of $$P$$ and $$Q$$ we have $$\begin{equation*} f_Q(\x) where $$\x=(x_1,\dots,x_n)\in\mathbb R^n$$ and $$|\cdot|$$ denotes the Euclidean norm. Hence, $$\begin{equation*} d_{TV}(P,Q)=p_1-p_2, \tag{1} \end{equation*}$$ where $$\begin{equation*} p_1:=P(S_n $$\begin{equation*} S_n:=\sum_1^n U_i,\quad T_n:=S_{n-1}+V_n, \end{equation*}$$ $$\begin{equation*} U_i:=e^{\mu Z_i},\quad V_n:=e^{\mu(Z_n+\mu)}=e^{\mu^2}U_n, \end{equation*}$$ and the $$Z_i$$'s are iid standard normal random variables. Note that $$\begin{equation*} ES_n=na,\quad ET_n=na+O(1),\quad Var\, S_n=nb^2,\quad Var\,T_n=nb^2+O(1), \end{equation*}$$ where $$a:=e^{\mu^2/2}$$ and $$b^2:=e^{2\mu^2}-e^{\mu^2}$$. Also, $$E|U_1|^3<\infty$$. So, letting $$\Phi$$ denote the standard normal cumulative distribution function and using the Berry--Esseen inequality, we have $$\begin{equation*} p_1=P(S_n and $$\begin{multline*} p_2=P(T_n Thus, by (1), we have $$\begin{equation*} d_{TV}(P,Q)=O(1/\sqrt n), \end{equation*}$$ as conjectured.
• Very nice! I'll try to go through a similar approach for the general case with $\sigma \neq 1$ (or does your approach crucially rely on $\sigma=1$ somewhere?). It also seems that the Berry--Esseen inequality should let us get an explicit (rather than asymptotic) result involving $\mu, \sigma$ and $1/\sqrt{n}$, right? – Florian Tramèr Mar 15 '19 at 0:27
• I think this should work for $\sigma\ne1$ too, but I have not gone through the details. – Iosif Pinelis Mar 15 '19 at 2:16