2
$\begingroup$

Consider the following mixture model for a univariate density function $$ (1) \quad f(x)=\int_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2)) $$ where $D$ is a compact subset of $\mathbb{R}\times \mathbb{R}^+$, $(m, \sigma^2)$ denotes the pair of mean and variance, $g(\cdot; m, \sigma^2)$ is the univariate Normal density function with mean $m$ and variance $\sigma^2$, $\mu$ is a probability measure over $D$.

I'm interested in understanding which classes of density functions can (or cannot) be "approximated" as in (1).

That is, can we characterise the class, $\mathcal{F}$, of densities, $f(\cdot)$, for which there exists $D,\mu$ such that the "distance" between $f(\cdot)$ and $\int_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2))$ is small?

I found in various papers/books sentences along the lines of "There is an obvious sense in which the mixture of normals approach, given enough components, can approximate any density" (see here for instance). I also found some papers proposing formalisation of this sentence within the finite mixture case. See also here for a related question. However, I could not find anything formally dealing with the infinite mixture case as (1).

Would you have some references to suggest? If I set $D$ very large, wouldn't (1) encompass a quite general class of densities?

$\endgroup$

1 Answer 1

3
$\begingroup$

$\newcommand\R{\mathbb R}\newcommand\si{\sigma}\newcommand\ep{\varepsilon}\newcommand\de{\delta}$Any probability distribution on $\R$ can be approximated by a discrete probability distribution on $\R$. Any discrete probability distribution on $\R$ is a mixture of Dirac probability distributions on $\R$. The Dirac probability distribution supported at a point $a\in\R$ can be approximated by the normal distribution with a small variance centered at $a$.

Thus indeed, if the set $D$ is large enough, then any probability distribution on $\R$ can be approximated by a mixture of normal distributions $N(m,\sigma^2)$ with $(m,\si^2)\in D$.


Another way: any probability distribution $\nu$ on $\R$ can be approximated by the convolution $\nu*N(0,\si^2)$ of $\nu$ with the centered normal distribution $N(0,\si^2)$ with a small variance $\si^2$, and such a convolution is a mixture of normal distributions: \begin{equation*} \nu*N(0,\si^2)=\int_\R N(y,\si^2)\nu(dy). \tag{0} \end{equation*} If now the set $D$ is large enough to, say, contain a set of the form $C_\de\times(0,\ep)$, where $\ep\in(0,\infty)$, $\de$ is a small positive number, and the set $C_\de\subseteq\R$ is such that $\nu(C_\de)>1-\de$, then for small $\si^2\in(0,\ep)$ \begin{equation*} \nu\approx\nu*N(0,\si^2)\approx\int_{C_\de} N(y,\si^2)\nu(dy) =\int_D N(y,s^2)\mu(dy\times d(s^2)), \tag{1} \end{equation*} where $\mu(dy\times d(s^2)):=\nu(dy)1(y\in C_\de,s^2=\si^2)$ -- so that $\nu$ will be approximated by the mixture $\int_D N(y,s^2)\mu(dy\times d(s^2))$ of normal distributions $N(m,s^2)$ with $(m,s^2)\in D$.

Details: The approximate equalities in (1) can be understood as follows. Suppose that $\si_n^2\downarrow0$ and $\de_n\downarrow0$ (as $n\to\infty$). Then, by Slutsky's theorem, $\nu*N(0,\si_n^2)\to\nu$ weakly; that is, for any bounded continuous function $g\colon\R\to\R$ we have \begin{equation*} (\nu*N(0,\si_n^2))(g)\to\nu(g), \end{equation*} where $\nu(g):=\int_\R g\,d\nu$; this is how the first approximate equality in (1) can be understood.

Next, by (0), for any bounded continuous function $g\colon\R\to\R$ with $M:=\sup_{x\in\R}|g(x)|<\infty$ we have \begin{equation*} \begin{aligned} &\Big|(\nu*N(0,\si_n^2))(g)-\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\Big| \\ =&\Big|\int_{\R} N(y,\si_n^2)(g)\nu(dy)-\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\Big| \\ \le&\int_{\R\setminus C_{\de_n}} N(y,\si^2)(|g|)\nu(dy) \\ \le& M\nu(\R\setminus C_{\de_n})\le M\de_n\to0; \end{aligned} \end{equation*} this is how the second approximate equality in (1) can be understood.

So, for any bounded continuous function $g\colon\R\to\R$, \begin{equation*} \Big(\int_{C_{\de_n}} N(y,\si_n^2)\nu(dy)\Big)(g) =\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\to\nu(g). \end{equation*} That is, we have the weak convergence of the normal mixtures $\int_{C_{\de_n}} N(y,\si_n^2)\nu(dy)$ to $\nu$. (The equality in the latter display is an instance of the Fubini theorem.)

$\endgroup$
30
  • 1
    $\begingroup$ @TEX : Done ( I too, did not notice that $\mu$ was used to denote two different things). $\endgroup$ Commented Jul 8, 2021 at 14:33
  • 1
    $\begingroup$ @TEX : (i) The dimension reduction should only be welcome. Anyway, the mixture can be rewritten as an integral over entire $D$ (as is now shown), with zero weights of some/most mixture components. You can say we do integrate over the variance values, but w.r. to the Dirac measure on $(0,\infty)$ supported just at one point $\sigma^2$. $\endgroup$ Commented Jul 8, 2021 at 15:35
  • 1
    $\begingroup$ @TEX : (ii) Of course, since $D$ is compact, the tails of any mixture of the normal distributions $N(m,s^2)$ with $(m,s^2)\in D$ will be light, normal-like in the far-away tail zones close enough to $\pm\infty$ and hence usually not relatively close to the corresponding far-away tails of the approximated distribution $\nu$. But as far as the absolute (not relative) approximation error is concerned, the far-away tail probabilities for $\nu$ will be small and thus close to the corresponding tail probabilities for the approximating mixture, if $D$ is large enough. $\endgroup$ Commented Jul 8, 2021 at 15:46
  • 1
    $\begingroup$ @TEX : "any probability distribution $\nu$ on $\mathbb R$ can be approximated by the convolution $\nu*N(0,\sigma^2)$ of $\nu$ with the centered normal distribution $N(0,\sigma^2)$ with a small variance $\sigma^2$" -- This follows e.g. by Slutsky's theorem en.wikipedia.org/wiki/Slutsky%27s_theorem . The other facts used in this answer are similarly very basic. $\endgroup$ Commented Jul 8, 2021 at 15:52
  • 1
    $\begingroup$ @TEX : "how should I formalise the symbol ≈ in your answer?" -- In the sense of the convergence in distribution. Recall that in your post you did not specify "approximated". $\endgroup$ Commented Jul 8, 2021 at 15:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .