Question

I am interested in a measure for the quality of fit to a curve which would distinguish the two cases shown in the following image (without addressing the fact that incidentally the right one has more samples, but rather addressing the way it 'determines' the curve in a better way): For the sake of it let's say I want to fit a parabola to this data. Is there some well known measure for this?

Answer 1

If you use the least square fit, the second case may have a better conditioned matrix but this measure may be hard to compute in practice. Still, it is going to be something in this venue because the story is not about the second curve being a better approximation than the first but about its being "more unique", so to say, and that is exactly what the condition number measures for solutions of linear systems.

Of course, since we are talking about approximate solutions, not exact ones, we may want to modify the notion of the condition number a bit. One possible quantity that seems relevant to the "approximate uniqueness" is the following: the least square problem is just about minimization of a quadratic form $Q(x)$ and if $y$ is the solution, then $Q(x)=Q(y)+(A(x-y),(x-y))$. Now, we want to see what is the penalty for going away from the optimal vector. So, both $\frac{\mbox{Tr\,}A}{Q(y)}$ and $\frac{\mu(A)}{Q(y)}$ where $\mu(A)$ is the least eigenvalue of $A$ seem to make sense as measures of such penalty. The higher is this number, the more unique is the approximation. The reason for the denominator is that I wanted to measure the sizes of deviations that change the minimum by certain percentage. You may want to do the absolute error instead, or something else. It may be a good idea to figure out what invariance properties you want from your measure first. For instance, should it be invariant with respect to stretchings or you think that two close points determine a line less precisely than two distant points?

Answer 2

If you use linear regression to calculate such parabolas, then the Pearson correlation coefficient is a good measure of the quality of the fit.

Answer 3

If I understand correctly, you would like to show that the right-hand dataset better defines the curve? There are many ways you could go about this, here's two:

A) You show that one dataset leaves less uncertainty about the parameters of the curve than the other. In a classical setting you'd fit the parameters using some kind of regression, and compare the error bars on the fitted parameters. In a Bayesian setting you could compare the two posterior distributions, maybe showing that one has lower posterior entropy than the other.

B) You show that one dataset leads to better predictive ability. Use cross-validation to compute an estimate of the generalisation error in each dataset, and compare that.