11
$\begingroup$

It seems that a mixture of Gaussians can approach any probability distribution, as the number of mixture components approaches infinity. Is this true? And if so, is it precise and correct to say that 'the Gaussian distribution is a dense subset of any distribution function'?

If not, how would you phrase this?

I have never seen anything about this in the literature about infinite mixtures of Gaussians.

Thanks very much!

$\endgroup$
2
  • $\begingroup$ Andrew, this doesn't answer your question, but is still on-topic. Brian Hayes just posted a nice blog post about disentangling mixtures of Gaussians. Take a look: bit-player.org/2010/disentangling-gaussians $\endgroup$ Jun 10, 2010 at 21:33
  • $\begingroup$ This follows from application of Glivenko-Cantelli and Slutsky's theorems. $\endgroup$ Apr 1, 2016 at 7:50

4 Answers 4

17
$\begingroup$

One way to say this is: Given any random variable $X$, there is a sequence of random variables $X_n$ whose distributions are finite mixtures of Gaussians, such that $X_n \Rightarrow X$ (i.e. $X_n$ converges to $X$ in distribution, or weakly). I don't believe you need to consider infinite mixtures per se.

It is true. First, note that any constant random variable can certainly be approximated in distribution by Gaussians (just let the variance tend to 0, or maybe you already consider constants to be Gaussian). But any random variable can be approximated in distribution by a mixture of constants. (Approximate the cdf of $X$ by step functions.)

In functional analysis language, one might say that in the Banach space $\mathcal{M}(\mathbb{R})$ of finite signed (or complex) measures on $\mathbb{R}$, the weak-* closed convex hull of the Gaussian probability measures contains all the probability measures. Equivalently, the convex combinations (i.e. mixtures) of the Gaussian probability measures are weak-* dense in the probability measures.

$\endgroup$
2
  • 1
    $\begingroup$ Could you offer a reference for this fact? Thanks. $\endgroup$ Mar 20, 2016 at 4:14
  • 1
    $\begingroup$ @eric_kernfeld: I don't know of a specific reference. This is elementary enough that I think most authors wouldn't feel the need to provide one. $\endgroup$ Mar 20, 2016 at 5:31
6
$\begingroup$

Any mixture of Gaussians has a density, which limits then sense in which a statement like you want to make can be true. The statement you propose doesn't make sense (in part) since a distribution is not a set.

One statement which I believe is true is

Mixtures of Gaussian distributions are dense in the set of probability distributions, with respect to the weak topology.

(By "weak topology" I mean the probabilists' weak topology, also called the topology of convergence in distribution, the vague topology, and the weak-* topology.)

I haven't checked details, but this should follow since in this topology, finite linear combinations of point masses are dense, and point masses can be approximated by Gaussian distributions.

The corresponding statement about density with respect to total variation is false, since discrete distributions cannot be approximated by distributions with density in total variation.

$\endgroup$
1
  • 1
    $\begingroup$ Rereading my answer, I realize it's a bit confusing that I'm using the word density with two unrelated meanings in the same sentence. Oh, well. $\endgroup$ Jun 9, 2010 at 17:30
1
$\begingroup$

People do "density estimation" with Gaussian "kernel functions". I.e. convolve the empirical distribution (the finite discrete distribution of the points in your sample) with a Gaussian density. So they're trying to approximate the unknown population density with a weighted average of Gaussians.

That's a simple approach; it then gets far more subtle and sophisticated. It's a whole field of research. Run the terms I put in quotes above through google scholar and google books.

$\endgroup$
1
$\begingroup$

Nestoridis, Vassili; Schmutzhard, Sebastian; Stefanopoulos, Vangelis, Universal series induced by approximate identities and some relevant applications, J. Approx. Theory 163, No. 12, 1783-1797 (2011). ZBL1231.35038., https://core.ac.uk/download/pdf/82702597.pdf

This paper provides an in-depth treatment of the topic.

$\endgroup$
1
  • $\begingroup$ If $T$ is a distribution then $T_n = T \ast n e^{-\pi n^2 x^2}$ is smooth, if $T$ was a probability density (ie. non-negative with $<T,1> =1$) then $T_n$ is a probability density so $T_n$ is smooth and $L^1$ and we can approximate $T_n \ast n e^{-\pi n^2 x^2}$ with the mixture of gaussian $\sum_{k=-K}^K \frac1K T_n(k/K) n e^{-\pi n^2 (x-k/K)^2}$. Finally $T_n \ast n e^{-\pi n^2 x^2}$ is an approximation of $T$. $\endgroup$
    – reuns
    Jul 27, 2019 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.