About a mixture Consider the following mixture model for a univariate density function
$$
(1) \quad f(x)=\int_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2))
$$
where $D$ is a compact subset of $\mathbb{R}\times \mathbb{R}^+$, $(m, \sigma^2)$ denotes the pair of mean  and variance, $g(\cdot; m, \sigma^2)$ is the univariate Normal density function
with mean $m$ and variance $\sigma^2$, $\mu$ is a probability measure over $D$.
I'm interested in understanding which classes of density functions can (or cannot) be "approximated" as in (1).
That is, can we characterise the class, $\mathcal{F}$, of densities, $f(\cdot)$, for which there exists $D,\mu$ such that the "distance" between $f(\cdot)$ and $\int_{(m, \sigma^2)\in D} g(x; m, \sigma^2) \mu(d(m, \sigma^2))$ is small?
I found in various papers/books  sentences along the lines of "There is an obvious sense in which the mixture of normals approach, given enough components, can approximate any density" (see here for instance).
I also found some papers proposing formalisation of this sentence within the finite mixture case. See also here for a related question. However, I could not find anything formally dealing with the infinite mixture case as (1).
Would you have some references to suggest? If I set $D$ very large, wouldn't (1) encompass a quite general class of densities?
 A: $\newcommand\R{\mathbb R}\newcommand\si{\sigma}\newcommand\ep{\varepsilon}\newcommand\de{\delta}$Any probability distribution on $\R$ can be approximated by a discrete probability distribution on $\R$. Any discrete probability distribution on $\R$ is a mixture of Dirac probability distributions on $\R$. The Dirac probability distribution supported at a point $a\in\R$ can be approximated by the normal distribution with a small variance centered at $a$.
Thus indeed, if the set $D$ is large enough, then any probability distribution on $\R$ can be approximated by a mixture of normal distributions $N(m,\sigma^2)$ with $(m,\si^2)\in D$.

Another way: any probability distribution $\nu$ on $\R$ can be approximated by the convolution $\nu*N(0,\si^2)$ of $\nu$ with the centered normal distribution $N(0,\si^2)$ with a small variance $\si^2$, and such a convolution is a mixture of normal distributions:
\begin{equation*}
    \nu*N(0,\si^2)=\int_\R N(y,\si^2)\nu(dy). \tag{0}
\end{equation*}
If now the set $D$ is large enough to, say, contain a set of the form $C_\de\times(0,\ep)$, where $\ep\in(0,\infty)$, $\de$ is a small positive number, and the set $C_\de\subseteq\R$ is such that $\nu(C_\de)>1-\de$, then for small $\si^2\in(0,\ep)$
\begin{equation*}
    \nu\approx\nu*N(0,\si^2)\approx\int_{C_\de} N(y,\si^2)\nu(dy)
=\int_D N(y,s^2)\mu(dy\times d(s^2)), \tag{1}
\end{equation*}
where $\mu(dy\times d(s^2)):=\nu(dy)1(y\in C_\de,s^2=\si^2)$ --
so that $\nu$ will be approximated by the mixture $\int_D N(y,s^2)\mu(dy\times d(s^2))$ of normal distributions $N(m,s^2)$ with $(m,s^2)\in D$.
Details: The approximate equalities in (1) can be understood as follows. Suppose that $\si_n^2\downarrow0$ and $\de_n\downarrow0$ (as $n\to\infty$). Then, by Slutsky's theorem, $\nu*N(0,\si_n^2)\to\nu$ weakly; that is, for any bounded continuous function $g\colon\R\to\R$ we have
\begin{equation*}
    (\nu*N(0,\si_n^2))(g)\to\nu(g), 
\end{equation*}
where $\nu(g):=\int_\R g\,d\nu$; this is how the first approximate equality in (1) can be understood.
Next, by (0), for any bounded continuous function $g\colon\R\to\R$ with $M:=\sup_{x\in\R}|g(x)|<\infty$ we have
\begin{equation*}
\begin{aligned}
    &\Big|(\nu*N(0,\si_n^2))(g)-\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\Big| \\ 
    =&\Big|\int_{\R} N(y,\si_n^2)(g)\nu(dy)-\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\Big| \\ 
    \le&\int_{\R\setminus C_{\de_n}} N(y,\si^2)(|g|)\nu(dy) \\ 
    \le& M\nu(\R\setminus C_{\de_n})\le M\de_n\to0; 
\end{aligned}
\end{equation*}
this is how the second approximate equality in (1) can be understood.
So, for any bounded continuous function $g\colon\R\to\R$,
\begin{equation*}
    \Big(\int_{C_{\de_n}} N(y,\si_n^2)\nu(dy)\Big)(g)
=\int_{C_{\de_n}} N(y,\si_n^2)(g)\nu(dy)\to\nu(g). 
\end{equation*}
That is, we have the weak convergence of the normal mixtures $\int_{C_{\de_n}} N(y,\si_n^2)\nu(dy)$ to $\nu$. (The equality in the latter display is an instance of the Fubini theorem.)
