# Mixtures of Gaussian distributions dense in distributions?

It seems that a mixture of Gaussians can approach any probability distribution, as the number of mixture components approaches infinity. Is this true? And if so, is it precise and correct to say that 'the Gaussian distribution is a dense subset of any distribution function'?

If not, how would you phrase this?

Thanks very much!

-
Andrew, this doesn't answer your question, but is still on-topic. Brian Hayes just posted a nice blog post about disentangling mixtures of Gaussians. Take a look: bit-player.org/2010/disentangling-gaussians –  Tom LaGatta Jun 10 '10 at 21:33

One way to say this is: Given any random variable $X$, there is a sequence of random variables $X_n$ whose distributions are finite mixtures of Gaussians, such that $X_n \Rightarrow X$ (i.e. $X_n$ converges to $X$ in distribution, or weakly). I don't believe you need to consider infinite mixtures per se.

It is true. First, note that any constant random variable can certainly be approximated in distribution by Gaussians (just let the variance tend to 0, or maybe you already consider constants to be Gaussian). But any random variable can be approximated in distribution by a mixture of constants. (Approximate the cdf of $X$ by step functions.)

In functional analysis language, one might say that in the Banach space $\mathcal{M}(\mathbb{R})$ of finite signed (or complex) measures on $\mathbb{R}$, the weak-* closed convex hull of the Gaussian probability measures contains all the probability measures. Equivalently, the convex combinations (i.e. mixtures) of the Gaussian probability measures are weak-* dense in the probability measures.

-

Any mixture of Gaussians has a density, which limits then sense in which a statement like you want to make can be true. The statement you propose doesn't make sense (in part) since a distribution is not a set.

One statement which I believe is true is

Mixtures of Gaussian distributions are dense in the set of probability distributions, with respect to the weak topology.

(By "weak topology" I mean the probabilists' weak topology, also called the topology of convergence in distribution, the vague topology, and the weak-* topology.)

I haven't checked details, but this should follow since in this topology, finite linear combinations of point masses are dense, and point masses can be approximated by Gaussian distributions.

The corresponding statement about density with respect to total variation is false, since discrete distributions cannot be approximated by distributions with density in total variation.

-
Rereading my answer, I realize it's a bit confusing that I'm using the word density with two unrelated meanings in the same sentence. Oh, well. –  Mark Meckes Jun 9 '10 at 17:30

People do "density estimation" with Gaussian "kernel functions". I.e. convolve the empirical distribution (the finite discrete distribution of the points in your sample) with a Gaussian density. So they're trying to approximate the unknown population density with a weighted average of Gaussians.

That's a simple approach; it then gets far more subtle and sophisticated. It's a whole field of research. Run the terms I put in quotes above through google scholar and google books.

-