Non-parametric regression and curvature

Question

Given a finite set of points $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$ in the plane, Linear Regression tells us how to find the straight line "$y=a+bx$" best approximating the given points, in the sense that the quantity $$ E(a, b)= \sum_{i=1}^n\big (ax_i+b-y_i\big )^2 $$ is as small as possible. However, when the given points are believed to be generated by a nonlinear phenomenon, perhaps the time series of an exponential process, one might prefer to replace "$y=a+bx$" with some other class of functions, often one that is parametrized by a small number of parameters, in which case one is often interested in finding the values of such parameters that minimize some sensible error estimate replacing our $E(a, b)$ above.

On the other hand, according to the Wikipedia, Nonparametric Statistics is the branch of Statistics that is not based solely on parametrized models, although the term non-parametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance.

I believe that one of the reasons for the above disclaimer is that, should one adopt a completely non-parametric approach for fitting a function to a given set of points $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$, as above, there will be too much freedom and hence the problem will become too easy (a piecewise linear function joining the points being a trivial solution), while I think it is safe to assume that such a solution will likely shed no light on the phenomenon under study.

In order to avoid such trivialities one must therefore either choose a model (parametrized family of functions) beforehand or else impose extra conditions on the fitted function. One possible approach is to require that the fitted function should not wiggle too much and, since the curvature of the graph of a function is related to its second derivative, a possible measure of wiggleness could be taken as $$ W(f) = \int_a^bf''(x)^2\, d(x). $$ Note that, if $W(f)=0$, then $f$ is necessarily a straight line, which certainly does not wiggle at all.

Question: Given a finite set of points $(x_1, y_1), (x_2, y_2),..., (x_n, y_n)$ in the plane, such that the $x_i$ all lie in the interval $[a, b]$, does there exist a twice differentiable function $f$ defined on $[a, b]$, such that the quantity $$L(f) = \sum_{i=1}^n\big (f(x_i)-y_i\big )^2 + \int_a^bf''(x)^2\, d(x)$$
is minimum among all such functions? In other words, does the functional $L$ defined above attain a minimum on $C^2([a,b])$?

Answer 1

Yes, that is the cubic smoothing spline with $\lambda$ (multiplier on the integral, which controls the amount of smoothing) ) = 1. See https://en.wikipedia.org/wiki/Smoothing_spline#Cubic_spline_definition .

Answer 2

I may start answering by pointing out that the term "nonparametrics statistics" is essentially "parametric". The existing methods (e.g. Smoothing splines) in nonparametrics, are somehow all parametrized by some (finite dimensional) set of parameters.

The term "flexible" is true. However, from a applied perspective, you need to conduct a model selection to choose a fixed parameter space to do statistical inference. Alternatively, if you go for a Bayesian nonparametric modeling, instead of model selection, people usually will do model averaging (e.g. RJ-MCMC used for Bayesian modeling)

...there will be too much freedom and hence the problem will become too easy (a piecewise linear function joining the points being a trivial solution), while I think it is safe to assume that such a solution will likely shed no light on the phenomenon under study...

Again, let's consider the smoothing splines. The main restriction we attempt to impose is "knot conditions" that lead to certain order of smoothness at certain sub-domains. Once you put these restraints and choose the splines as basis, the linearity arise from the space of these basis functions. Therefore, "parametric linear" statistical inference like ANOVA can again be conducted. The answer to your question can also be found in the same book (or following wikipdeia's answer here).

In a more general sense, you can do some probabilistic inference without assuming linearity, for example, in a Banach space (e.g. Probability in Banach Space). But in that case, most object you derive will not have specific forms like splines.