Goodness-of-fit test for Generalized Additive Model

Question

Hi,

I would like to do this:

fit <- test( measured_values, fitted_values )

Where:

• the return value from the test function is: 0 < fit < 1.
• measured_values are the observed data.
• fitted_values are the data for the curve produced by GAM for the measured_values.

What test can I use to compare the data sets that will result in a number between 0 and 1, where 0 indicates the measured values and the fitted values are not a good fit and 1 indicates the fitted values fit perfectly to the measured values?

For example, consider the data plotted here: https://i.sstatic.net/wcrey.jpg

The fit, produced by GAM, is fairly close to ideal. However, the standard correlations (shown in the bottom left) do not accurately indicate the goodness of fit.

Thank you.

Answer 1

I meant only what is usually meant and you're being completely cryptic.

A wikipedia article titled "generalized additive model" says you've got $$ g(\operatorname{E}(Y))=\beta_0 + f_1(x_1) + f_2(x_2)+ \cdots + f_m(x_m) $$ and then says "The functions $f_i(x_i)$ may be fit using parametric or non-parametric means, thus providing the potential for better fits to data than other methods. The method hence is very general". In other words, there's a lot you haven't said!! And then "a typical GAM might use a scatterplot smoothing function such as a locally weighted mean for $f_1(x_1)$, and then use a factor model for $f_2(x_2)$". So again, you're not telling us how you got this fit? E.g. are you using least squares? Or maximum likelihood? Are you picking the function $f_1$ from some parametrized or otherwise well-behave class of functions? If so, WHICH ONE? And what function are you using for $g$? You're making me guess what you mean. Your graph looked as if your value of $m$ was probably 1. Which makes me begin to wonder why you're calling it "additive". I'm guessing the graph you linked to was not brought down from Heaven by an archangel. Can you say how you got it?

Answer 2

The way ANOVA usually works, you have a sum of squares $\sum_i(y_i - \overline{y})^2$ where $\overline{y}$ is the mean $\sum_i y_i /n$ giving the amount of variability in the $y$-values, and you partition it into various (at least two) components, one of which is the sum of squares of residuals, which is the "unexplained" sum of squares. You also have an "explained" sum of squares, and the ratio of the two is often used as a test statistic. Which hypothesis is being tested is more than I want to get into before I ascertain that we're communicating.

You may want to look at this: http://en.wikipedia.org/wiki/Lack-of-fit_sum_of_squares

That fits right into the "ANOVA" context.

One thing you don't get from this is the probability that the model fits. That's something you might get from a Bayesian method, but ANOVA as usually understood is a frequentist method.