I am interested in the relation between a statistical model $(\Omega, \mathcal{F}, (\mathbb{P}^\theta : \theta\in\Theta))$, where the hypotheses are "$\mathbb{P}^\theta$ is a good approximation of the real probability density" and statistical learning theory, where given input data $X:\Omega \rightarrow \mathbb{R}^m$ and labels $Y:\Omega \rightarrow \mathbb{R}^n$ we search for a function $f$ minimizing the risk $$\mathcal{E}(f)=\int_\Omega (f(X(\omega)) - Y(\omega))^2 ~\text{d}\mathbb{P}(\omega),$$ leading to hypotheses "$f$ is a good approximation of the relation between X and Y"

Are the two approaches equivalent? Can I reformulate the statistical learning approach to a statistical model, thus obtaining a set of probability measures $\mathbb{P}_f$ on some probability space, or are there fundamental differences?

I haven't found a resource where they rigorously actually do this. Any help or references are well appreciated.


To be able to reformulate what you call the statistical learning approach, you need to make some assumptions on the joint law $(X,Y)$, and how it is related to the function $f$. For instance, you may assume that the following relation holds (regression model): $$ Y = f(X) + \varepsilon,$$ where $f$ belongs to some family $F$ of functions (parameterized or not), and $\varepsilon$ is independent from $X$, with some assumptions on its law (typically that it is centered, with a prescribed variance). Then, calling $\mathbb{P}_f$ the law of the couple $(X,Y)$, and assuming some measurability, the minimizer $\hat{f}$ of $\mathcal{E}(f)$ on $F$ is an estimator of $f$ in the statistical model $(\Omega, \mathcal{F},(\mathbb{P}_f,\ f\in F))$. Note that depending on the assumptions made on the class $F$, this estimator might or might not have good statistical properties.

If you want more information, you should probably check for the keyword "nonparametric regression".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.