2
$\begingroup$

Consider the following mixture model for a univariate density function $$ (1) \quad f(x)=\int_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2)) $$ where $D$ is a compact subset of $\mathbb{R}\times \mathbb{R}^+$, $(m, \sigma^2)$ denotes the pair of mean and variance, $g(\cdot; m, \sigma^2)$ is the univariate Normal density function with mean $m$ and variance $\sigma^2$, $\mu$ is a probability measure over $D$.

I'm interested in understanding which classes of density functions can (or cannot) be "approximated" as in (1).

That is, can we characterise the class, $\mathcal{F}$, of densities, $f(\cdot)$, for which there exists $D,\mu$ such that the "distance" between $f(\cdot)$ and $\int_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2))$ is small?

I found in various papers/books sentences along the lines of "There is an obvious sense in which the mixture of normals approach, given enough components, can approximate any density" (see here for instance). I also found some papers proposing formalisation of this sentence within the finite mixture case. See also here for a related question. However, I could not find anything formally dealing with the infinite mixture case as (1).

Would you have some references to suggest? If I set $D$ very large, wouldn't (1) encompass a quite general class of densities?

$\endgroup$

1 Answer 1

3
$\begingroup$

$\newcommand\R{\mathbb R}\newcommand\si{\sigma}\newcommand\ep{\varepsilon}\newcommand\de{\delta}$Any probability distribution on $\R$ can be approximated by a discrete probability distribution on $\R$. Any discrete probability distribution on $\R$ is a mixture of Dirac probability distributions on $\R$. The Dirac probability distribution supported at a point $a\in\R$ can be approximated by the normal distribution with a small variance centered at $a$.

Thus indeed, if the set $D$ is large enough, then any probability distribution on $\R$ can be approximated by a mixture of normal distributions $N(m,\sigma^2)$ with $(m,\si^2)\in D$.


Another way: any probability distribution $\nu$ on $\R$ can be approximated by the convolution $\nu*N(0,\si^2)$ of $\nu$ with the centered normal distribution $N(0,\si^2)$ with a small variance $\si^2$, and such a convolution is a mixture of normal distributions: \begin{equation*} \nu*N(0,\si^2)=\int_\R N(y,\si^2)\nu(dy). \tag{0} \end{equation*} If now the set $D$ is large enough to, say, contain a set of the form $C_\de\times(0,\ep)$, where $\ep\in(0,\infty)$, $\de$ is a small positive number, and the set $C_\de\subseteq\R$ is such that $\nu(C_\de)>1-\de$, then for small $\si^2\in(0,\ep)$ \begin{equation*} \nu\approx\nu*N(0,\si^2)\approx\int_{C_\de} N(y,\si^2)\nu(dy) =\int_D N(y,s^2)\mu(dy\times d(s^2)), \tag{1} \end{equation*} where $\mu(dy\times d(s^2)):=\nu(dy)1(y\in C_\de,s^2=\si^2)$ -- so that $\nu$ will be approximated by the mixture $\int_D N(y,s^2)\mu(dy\times d(s^2))$ of normal distributions $N(m,s^2)$ with $(m,s^2)\in D$.

Details: The approximate equalities in (1) can be understood as follows. Suppose that $\si_n^2\downarrow0$ and $\de_n\downarrow0$ (as $n\to\infty$). Then, by Slutsky's theorem, $\nu*N(0,\si_n^2)\to\nu$ weakly; that is, for any bounded continuous function $g\colon\R\to\R$ we have \begin{equation*} (\nu*N(0,\si_n^2))(g)\to\nu(g), \end{equation*} where $\nu(g):=\int_\R g\,d\nu$; this is how the first approximate equality in (1) can be understood.

Next, by (0), for any bounded continuous function $g\colon\R\to\R$ with $M:=\sup_{x\in\R}|g(x)|<\infty$ we have \begin{equation*} \begin{aligned} &\Big|(\nu*N(0,\si_n^2))(g)-\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\Big| \\ =&\Big|\int_{\R} N(y,\si_n^2)(g)\nu(dy)-\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\Big| \\ \le&\int_{\R\setminus C_{\de_n}} N(y,\si^2)(|g|)\nu(dy) \\ \le& M\nu(\R\setminus C_{\de_n})\le M\de_n\to0; \end{aligned} \end{equation*} this is how the second approximate equality in (1) can be understood.

So, for any bounded continuous function $g\colon\R\to\R$, \begin{equation*} \Big(\int_{C_{\de_n}} N(y,\si_n^2)\nu(dy)\Big)(g) =\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\to\nu(g). \end{equation*} That is, we have the weak convergence of the normal mixtures $\int_{C_{\de_n}} N(y,\si_n^2)\nu(dy)$ to $\nu$. (The equality in the latter display is an instance of the Fubini theorem.)

$\endgroup$
28
  • 1
    $\begingroup$ @TEX : Done ( I too, did not notice that $\mu$ was used to denote two different things). $\endgroup$ Jul 8, 2021 at 14:33
  • 1
    $\begingroup$ @TEX : (i) The dimension reduction should only be welcome. Anyway, the mixture can be rewritten as an integral over entire $D$ (as is now shown), with zero weights of some/most mixture components. You can say we do integrate over the variance values, but w.r. to the Dirac measure on $(0,\infty)$ supported just at one point $\sigma^2$. $\endgroup$ Jul 8, 2021 at 15:35
  • 1
    $\begingroup$ @TEX : (ii) Of course, since $D$ is compact, the tails of any mixture of the normal distributions $N(m,s^2)$ with $(m,s^2)\in D$ will be light, normal-like in the far-away tail zones close enough to $\pm\infty$ and hence usually not relatively close to the corresponding far-away tails of the approximated distribution $\nu$. But as far as the absolute (not relative) approximation error is concerned, the far-away tail probabilities for $\nu$ will be small and thus close to the corresponding tail probabilities for the approximating mixture, if $D$ is large enough. $\endgroup$ Jul 8, 2021 at 15:46
  • 1
    $\begingroup$ @TEX : "any probability distribution $\nu$ on $\mathbb R$ can be approximated by the convolution $\nu*N(0,\sigma^2)$ of $\nu$ with the centered normal distribution $N(0,\sigma^2)$ with a small variance $\sigma^2$" -- This follows e.g. by Slutsky's theorem en.wikipedia.org/wiki/Slutsky%27s_theorem . The other facts used in this answer are similarly very basic. $\endgroup$ Jul 8, 2021 at 15:52
  • 1
    $\begingroup$ @TEX : "how should I formalise the symbol ≈ in your answer?" -- In the sense of the convergence in distribution. Recall that in your post you did not specify "approximated". $\endgroup$ Jul 8, 2021 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.