0
$\begingroup$

Suppose that $X_1,..,X_n$ are i.i.d real random variables with density $f \in L_2(\mathbb R)$, and that $g_i$ are function forming an orthonormal basis of $L_2(\mathbb R)$, i.e :

$$f(x) = \sum\limits_{i} a_i g_i(x) \text{ for } a_i = \int g_i(x) f(x) dx$$

Set the Monte-Carlo coefficients to be $\widehat{a_i} = \frac{1}{n} \sum\limits_{i=1}^{n} g_i(X_i)$, and denote $\hat{f}(x) = \sum\limits_{i} \widehat{a_i} g_i(x)$.

I want to show that: $$\lim\limits_{n \to \infty} \lVert f - \hat{f} \rVert_{2}^2 = \lim\limits_{n \to \infty} \sum\limits_{i} (a_i - \widehat{a_i})^2 = 0$$

I am able to show that the estimators $\widehat{a_i}$ are unbiaised and converge correctly to $a_i$, but here I need some kind of uniform convergence, right ?

$\endgroup$

2 Answers 2

1
$\begingroup$
  1. The (more) correct definition of the $\widehat{a_i}$'s should be $$\widehat{a_{n,i}}:=\frac1n\,\sum_{j=1}^n g_i(X_j).$$

  2. So, $$\widehat{a_{n,i}}=\int_{\mathbb R}\mu_n(t) g_i(t)\,dt,$$ where $$\mu_n(t):=\frac1n\,\sum_{j=1}^n \delta_{X_j}(t)$$ and $\delta_x$ is the Dirac probability measure at $x$, viewed as the (say) Schwartz distribution. So, the $\widehat{a_{n,i}}$'s may be viewed as the "coordinates" of the Schwartz distribution $\mu_n$ in the basis $(g_i)$ of $L_2(\mathbb R)$. So, if we had $\sum_i \widehat{a_{n,i}}^2<\infty$, we would also have $\mu_n\in L_2(\mathbb R)$, which is of course false. So, $\sum_i \widehat{a_{n,i}}^2=\infty$ for any basis $(g_i)$ of $L_2(\mathbb R)$, and hence $\sum_i (a_i-\widehat{a_{n,i}})^2=\infty$ for any real $a_i$'s such that $\sum_i a_i^2<\infty$.

  3. Another view at why your idea of estimation of the density $f$ did not succeed is that the sequence $(\mu_n)$ of the (empirical) probability measures converges to the probability measure (say $\mu$) with density $f$ only weakly, and the probability measure $\mu_n$ does not even have a density, to converge to $f$ in any sense or to do anything else.

  4. Generally, it appears unnatural to estimate a pdf in an $L^2$ framework. The natural framework should be $L^1$. This is the view advocated (I think persuasively) by some authors, including Devroye and Gyorfi, who write that their book "develops, from first principles, the ``natural'' theory for density estimation, L1, and shows why the classical L2 theory masks some fundamental properties of density estimates".

$\endgroup$
5
  • $\begingroup$ Thanks for the extensive explanation. So, by your point 3., am i right to say that $\widehat{a_i}$ converges weakly to $a_i$ ? Which means that for all $i$, $(a_i - \widehat{a_i})^2 \to 0$. Then, why the fuck does the sum for all $i$ not converge... Dont get me wrong, I got your argument, it convinced me, but I still do not feel it. $\endgroup$
    – lrnv
    Jan 29, 2021 at 16:26
  • $\begingroup$ @lrnv : (i) It does not make sense to say that $\widehat{a_i}$ converges weakly to $a_i$. However, it is true that $\widehat{a_{n,i}}\to a_i$ (as $n\to\infty$) if the function $g_i$ is bounded and continuous; this follows because the empirical probability measure $\mu_n$ converges weakly to the probability measure $\mu$ with density $f$. (ii) If $b_{n,i}\to b_i$ for each $i$, this does not in general imply that $\sum_i b_{n,i}\to\sum_i b_i$; e.g. consider $b_{n,i}:=1(n=i)$ and $b_i:=0$. $\endgroup$ Jan 29, 2021 at 16:41
  • $\begingroup$ Again, thanks for the details. So to clarify, $\widehat{a_i} -> a_i$ but there is no way of making $\lVert f - \hat{f} \rVert_2^2$ go to 0 ? (according to point 2.). It it still strenge to me, as $\mathbb E( \lVert f - \hat f \rVert_2^2) \to 0$ and $\lVert f - \hat f \rVert_2$ is always positive.. $\endgroup$
    – lrnv
    Jan 29, 2021 at 16:48
  • $\begingroup$ @lrnv : You have $\sum_i \widehat{a_{n,i}}^2=\infty$. So, the very definition $\hat f(x): = \sum_i\widehat{a_i} g_i(x)$ makes no sense. So, you don't even have a $\hat f$. $\endgroup$ Jan 29, 2021 at 16:55
  • $\begingroup$ Yes you are right, I see it now. $\endgroup$
    – lrnv
    Jan 29, 2021 at 17:03
1
$\begingroup$

I think the sum on the rhs is not generally finite. Take the case where the $g_i(x) = $ ith Walsh function ( which satisfy $g_i^2 = 1$ ) and only one of the $a_i < \infty$ , n=1, and $f$ is uniform. Then you are summing $\Sigma g_i^2(X_1)$. I've left out the 1 non zero coefficient, but it doesn't change the point. I think the same issue occurs with larger n.

$\endgroup$
4
  • $\begingroup$ I'm not quite getting your point. I started by saying that $f$ belong to L_2, which means that $f$ converges. If i read you correctly, you propose a case where coefficients are infinite, which means that the function is not even analytic, and certainly not square-integrable. So this is different. Did i read you correctly ? $\endgroup$
    – lrnv
    Jan 29, 2021 at 12:53
  • $\begingroup$ Never mind my previous comment, now i get what you mean. Ok the walsh basis is quite peculiar. Maybe we could restrict to basis of continuous functions ? The functions in my basis are even integrable, derivable, etc.. What i mean is: Is there some condition on the basis we could express that would make my assumption true ? EDIT: Fundamentally, i dont think i am summing the squares of the basis functions... I'm summing the square distances between them and the empirical ones ! $\endgroup$
    – lrnv
    Jan 29, 2021 at 13:01
  • $\begingroup$ I think you can't take the obvious estimate for f, it's probably not an L^2 function, and you probably need to truncate the sum ($ \hat{f}(x) = \sum\limits_{i}^{u(n)} \widehat{a_i} g_i(x)$ where u(n) some upper bound) to get a good estimate. I believe I've seen this issue, but don't recall the details. $\endgroup$
    – mike
    Jan 29, 2021 at 14:16
  • $\begingroup$ For my problem i do need the full sum, although yes people usually truncate the estimator. So there is no convergence at all that i an obtain for these kind of estimators ? $\endgroup$
    – lrnv
    Jan 29, 2021 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.