# Distance between distributions and distance of moments

Let's say I have a sequence of random variables $X_n$ such that $$\mathbf E X_n^k = \mathbf E X^k+O(a_k/\sqrt{n})\quad\text{for all }k\in\mathbb N,\tag{\ast}$$ where $X$ is a random variable of standard (zero mean, unit variance) Gaussian distribution and $a_k$ are some constants which typically grow with $k$ (for my purposes they would be $a_k=(k/2)!$, for example).

Since the normal distribution is uniquely determined by its moments $(\ast)$ implies that $X_n\Rightarrow X$ weakly, as $n\to\infty$.

Is there an appropriate way to formalize the distance $d$ between the distributions of $X_n$ and $X$, such that $(\ast)$ implies $$d(X_n,X)=O(f(n))$$ for some function $f$? I think $(\ast)$ is not strong enough to control distance measures like the Kullback-Leibler divergence or the Hellinger distance, but there might be some appropriate weaker notion?

I think that $(\ast)$ implies $$\mathbf E f(X_n)=\mathbf E f(X)+O(1/\sqrt{n})$$ for a small class of test functions $f$, but depending on the growth of $a_k$, this class might be vary small.

• How about the sup of modulus of the difference of characteristic functions over some complex disc around the origin? It's not a metric on the whole space of probability measures but on a space that seems big enough for your purposes. Nov 8, 2016 at 23:43

The natural thing to compare in this context seems to be the moment generating functions of $X_n$ and $X$. In particular, consider: \begin{align*} \mathbf{E} \exp(t X_n) - \mathbf{E} \exp(t X) &= \sum_{k=0}^{\infty} \frac{t^k}{k!} \left( \mathbf{E} X_n^k - \mathbf{E} X^k \right) \\ &\le \frac{1}{\sqrt{n}} \sum_{k=0}^{\infty} \frac{t^k}{k!} (k/2)! = \frac{1}{\sqrt{n}} \left( 1 + e^{t^2/4} \sqrt{\pi} t + e^{t^2/4} \sqrt{\pi} t \operatorname{Erf}(t/2) \right) \end{align*} where we used the hypothesis given by the OP.

Given this bound on their moment generating functions (or a similar bound on the characteristic function), to what extent do the laws of $X_n$ and $X$ agree? There seems to be an interesting discussion about this in the statistics literature.

• McCullagh, Peter. "Does the moment-generating function characterize a distribution?" The American Statistician 48.3 (1994): 208-208.
• Waller, Lance A. "Does the characteristic function numerically distinguish distributions?" The American Statistician 49.2 (1995): 150-152.
• Luceño, Alberto. "Further evidence supporting the numerical usefulness of characteristic functions." The American Statistician 51.3 (1997): 233-234.
• I was not aware about this discussion in the statistics literature. Especially the McCullagh paper is somewhat surprising. Thanks for the interesting references! Nov 9, 2016 at 9:12
• A further comment: the above bound implies that the Lévy distance between $X_n$ and $X$ can be bounded by $c\frac{\log\log n}{\sqrt{\log n}}$ for some constant $c$. Nov 10, 2016 at 13:46
• @Julian A characterization of the Lévy distance I am familiar with is the one given in Problem 26.15 of the 4th edition of Probability and Measure by P. Billingsley. However, I don't see how to combine the above bound and this characterization to obtain this result. Could you please clarify? Nov 10, 2016 at 14:55
• This follows from the inequality $d_L(X_n,X)\leq \frac{1}{\pi}\int_{0}^T \lvert \frac{\phi_n(t)-\phi(t)}{t}\rvert dt + \frac{2e\log T}{T}$ which holds for $T\geq 2$ and where $\phi_n, \phi$ are the characteristic functions of $X_n,X$. The above form is from Theorem 1.4.13., N. G. Ushakov. Selected topics in characteristic functions. Walter de Gruyter, 2011 Nov 10, 2016 at 22:47

The classical approach to this is called "smoothing" and is described in Chapter 16 of Feller's Introduction to Probability Theory, Vol 2. Basically you use the bounds on the moment differences to bound the difference between the characteristic functions near the origin, then adjust a parameter (called $T$ by Feller) to get the best result.

I recommend a more general survey Proximity of probability distributions in terms of Fourier–Stieltjes transforms by S.G.Bobkov. Using various integral estimates, just like some mentioned in the comments, he derives several bounds for Kolmogorov distance, total variation distance, KL-divergence and so on.

Bounding distances to standard normal using characteristic functions falls into Berry-Essen inequalities. Theorem 2.1 in Bobkov's work, for the Kolmogorov–Smirnov distance $$d_{KS}$$ gives: $$d_{KS}(X_n,X) \leqslant O(1)\cdot \int_{-T}^{T}\left|\frac{\Phi_n(t)-\Phi(t)}{t}\right|\mbox{d} t + O(T^{-1}).$$ Under our assumptions (the series in the accepted answer shouldn't have a constant term!) $$|\Phi_n(t)-\Phi(t)| \leqslant O(n^{-1/2})\cdot t \mathrm{e}^{O(t^2)},$$ so we obtain $$d_{KS}(X_n,X) \leqslant O(n^{-1/2})\cdot \mathrm{e}^{O(T^2)} + O(T^{-1}),$$ and by optimizing over $$T$$ finally $$d_{KS}(X_n,X) \leqslant O\left(\frac{1}{\sqrt{\log n}}\right).$$