Convergence of an orthormal expansion of the density

Question

Suppose that $X_1,..,X_n$ are i.i.d real random variables with density $f \in L_2(\mathbb R)$, and that $g_i$ are function forming an orthonormal basis of $L_2(\mathbb R)$, i.e :

$$f(x) = \sum\limits_{i} a_i g_i(x) \text{ for } a_i = \int g_i(x) f(x) dx$$

Set the Monte-Carlo coefficients to be $\widehat{a_i} = \frac{1}{n} \sum\limits_{i=1}^{n} g_i(X_i)$, and denote $\hat{f}(x) = \sum\limits_{i} \widehat{a_i} g_i(x)$.

I want to show that: $$\lim\limits_{n \to \infty} \lVert f - \hat{f} \rVert_{2}^2 = \lim\limits_{n \to \infty} \sum\limits_{i} (a_i - \widehat{a_i})^2 = 0$$

I am able to show that the estimators $\widehat{a_i}$ are unbiaised and converge correctly to $a_i$, but here I need some kind of uniform convergence, right ?

Answer 1

1. The (more) correct definition of the $\widehat{a_i}$'s should be $$\widehat{a_{n,i}}:=\frac1n\,\sum_{j=1}^n g_i(X_j).$$

2. So, $$\widehat{a_{n,i}}=\int_{\mathbb R}\mu_n(t) g_i(t)\,dt,$$ where $$\mu_n(t):=\frac1n\,\sum_{j=1}^n \delta_{X_j}(t)$$ and $\delta_x$ is the Dirac probability measure at $x$, viewed as the (say) Schwartz distribution. So, the $\widehat{a_{n,i}}$'s may be viewed as the "coordinates" of the Schwartz distribution $\mu_n$ in the basis $(g_i)$ of $L_2(\mathbb R)$. So, if we had $\sum_i \widehat{a_{n,i}}^2<\infty$, we would also have $\mu_n\in L_2(\mathbb R)$, which is of course false. So, $\sum_i \widehat{a_{n,i}}^2=\infty$ for any basis $(g_i)$ of $L_2(\mathbb R)$, and hence $\sum_i (a_i-\widehat{a_{n,i}})^2=\infty$ for any real $a_i$'s such that $\sum_i a_i^2<\infty$.

3. Another view at why your idea of estimation of the density $f$ did not succeed is that the sequence $(\mu_n)$ of the (empirical) probability measures converges to the probability measure (say $\mu$) with density $f$ only weakly, and the probability measure $\mu_n$ does not even have a density, to converge to $f$ in any sense or to do anything else.

4. Generally, it appears unnatural to estimate a pdf in an $L^2$ framework. The natural framework should be $L^1$. This is the view advocated (I think persuasively) by some authors, including Devroye and Gyorfi, who write that their book "develops, from first principles, the ``natural'' theory for density estimation, L1, and shows why the classical L2 theory masks some fundamental properties of density estimates".

Answer 2

I think the sum on the rhs is not generally finite. Take the case where the $g_i(x) = $ ith Walsh function ( which satisfy $g_i^2 = 1$ ) and only one of the $a_i < \infty$ , n=1, and $f$ is uniform. Then you are summing $\Sigma g_i^2(X_1)$. I've left out the 1 non zero coefficient, but it doesn't change the point. I think the same issue occurs with larger n.