12
$\begingroup$

It seems that a mixture of Gaussians can approach any probability distribution, as the number of mixture components approaches infinity. Is this true? And if so, is it precise and correct to say that 'the Gaussian distribution is a dense subset of any distribution function'?

If not, how would you phrase this?

I have never seen anything about this in the literature about infinite mixtures of Gaussians.

Thanks very much!

Improve this question
$\endgroup$
2
  • $\begingroup$ Andrew, this doesn't answer your question, but is still on-topic. Brian Hayes just posted a nice blog post about disentangling mixtures of Gaussians. Take a look: bit-player.org/2010/disentangling-gaussians $\endgroup$
    – Tom LaGatta
    Commented Jun 10, 2010 at 21:33
  • $\begingroup$ This follows from application of Glivenko-Cantelli and Slutsky's theorems. $\endgroup$
    – Christian Chapman
    Commented Apr 1, 2016 at 7:50

4 Answers 4

Reset to default
18
$\begingroup$

One way to say this is: Given any random variable $X$, there is a sequence of random variables $X_n$ whose distributions are finite mixtures of Gaussians, such that $X_n \Rightarrow X$ (i.e. $X_n$ converges to $X$ in distribution, or weakly). I don't believe you need to consider infinite mixtures per se.

It is true. First, note that any constant random variable can certainly be approximated in distribution by Gaussians (just let the variance tend to 0, or maybe you already consider constants to be Gaussian). But any random variable can be approximated in distribution by a mixture of constants. (Approximate the cdf of $X$ by step functions.)

In functional analysis language, one might say that in the Banach space $\mathcal{M}(\mathbb{R})$ of finite signed (or complex) measures on $\mathbb{R}$, the weak-* closed convex hull of the Gaussian probability measures contains all the probability measures. Equivalently, the convex combinations (i.e. mixtures) of the Gaussian probability measures are weak-* dense in the probability measures.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Could you offer a reference for this fact? Thanks. $\endgroup$
    – eric_kernfeld
    Commented Mar 20, 2016 at 4:14
  • 1
    $\begingroup$ @eric_kernfeld: I don't know of a specific reference. This is elementary enough that I think most authors wouldn't feel the need to provide one. $\endgroup$
    – Nate Eldredge
    Commented Mar 20, 2016 at 5:31
6
$\begingroup$

Any mixture of Gaussians has a density, which limits then sense in which a statement like you want to make can be true. The statement you propose doesn't make sense (in part) since a distribution is not a set.

One statement which I believe is true is

Mixtures of Gaussian distributions are dense in the set of probability distributions, with respect to the weak topology.

(By "weak topology" I mean the probabilists' weak topology, also called the topology of convergence in distribution, the vague topology, and the weak-* topology.)

I haven't checked details, but this should follow since in this topology, finite linear combinations of point masses are dense, and point masses can be approximated by Gaussian distributions.

The corresponding statement about density with respect to total variation is false, since discrete distributions cannot be approximated by distributions with density in total variation.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Rereading my answer, I realize it's a bit confusing that I'm using the word density with two unrelated meanings in the same sentence. Oh, well. $\endgroup$
    – Mark Meckes
    Commented Jun 9, 2010 at 17:30
1
$\begingroup$

People do "density estimation" with Gaussian "kernel functions". I.e. convolve the empirical distribution (the finite discrete distribution of the points in your sample) with a Gaussian density. So they're trying to approximate the unknown population density with a weighted average of Gaussians.

That's a simple approach; it then gets far more subtle and sophisticated. It's a whole field of research. Run the terms I put in quotes above through google scholar and google books.

Improve this answer
$\endgroup$
1
$\begingroup$

Nestoridis, Vassili; Schmutzhard, Sebastian; Stefanopoulos, Vangelis, Universal series induced by approximate identities and some relevant applications, J. Approx. Theory 163, No. 12, 1783-1797 (2011). ZBL1231.35038., https://core.ac.uk/download/pdf/82702597.pdf

This paper provides an in-depth treatment of the topic.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If $T$ is a distribution then $T_n = T \ast n e^{-\pi n^2 x^2}$ is smooth, if $T$ was a probability density (ie. non-negative with $<T,1> =1$) then $T_n$ is a probability density so $T_n$ is smooth and $L^1$ and we can approximate $T_n \ast n e^{-\pi n^2 x^2}$ with the mixture of gaussian $\sum_{k=-K}^K \frac1K T_n(k/K) n e^{-\pi n^2 (x-k/K)^2}$. Finally $T_n \ast n e^{-\pi n^2 x^2}$ is an approximation of $T$. $\endgroup$
    – reuns
    Commented Jul 27, 2019 at 21:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .