# Goodness-of-fit test for Generalized Additive Model

Hi,

I would like to do this:

fit <- test( measured_values, fitted_values )


Where:

• the return value from the test function is: 0 < fit < 1.
• measured_values are the observed data.
• fitted_values are the data for the curve produced by GAM for the measured_values.

What test can I use to compare the data sets that will result in a number between 0 and 1, where 0 indicates the measured values and the fitted values are not a good fit and 1 indicates the fitted values fit perfectly to the measured values?

For example, consider the data plotted here: http://i.imgur.com/cFLRN.jpg

The fit, produced by GAM, is fairly close to ideal. However, the standard correlations (shown in the bottom left) do not accurately indicate the goodness of fit.

Thank you.

-
Could you be explicit about the model you're fitting? – Michael Hardy Jun 19 '10 at 20:55
@Michael: I do not understand your question; I thought the fit was the model. Essentially, I have two data sets and I want to see how closely they relate. In the chart, the green dots are measured temperature values and the fitted orange line is the Generalized Additive Model. I am trying to find a suitable estimator for the goodness-of-fit of the model to the measurements. – Dave Jarvis Jun 19 '10 at 21:16

I meant only what is usually meant and you're being completely cryptic.

A wikipedia article titled "generalized additive model" says you've got $$g(\operatorname{E}(Y))=\beta_0 + f_1(x_1) + f_2(x_2)+ \cdots + f_m(x_m)$$ and then says "The functions $f_i(x_i)$ may be fit using parametric or non-parametric means, thus providing the potential for better fits to data than other methods. The method hence is very general". In other words, there's a lot you haven't said!! And then "a typical GAM might use a scatterplot smoothing function such as a locally weighted mean for $f_1(x_1)$, and then use a factor model for $f_2(x_2)$". So again, you're not telling us how you got this fit? E.g. are you using least squares? Or maximum likelihood? Are you picking the function $f_1$ from some parametrized or otherwise well-behave class of functions? If so, WHICH ONE? And what function are you using for $g$? You're making me guess what you mean. Your graph looked as if your value of $m$ was probably 1. Which makes me begin to wonder why you're calling it "additive". I'm guessing the graph you linked to was not brought down from Heaven by an archangel. Can you say how you got it?

-
@Michael: The calculation uses R as follows: data.frame( x, fitted( gam( y ~ s(x) ) ) ), where x & y are years and measurements, respectively. The documentation notes that the method is iteratively reweighted least squares. There are no parameters. – Dave Jarvis Jun 20 '10 at 19:42
@Michael: I think I can use ANOVA and Chi-squared. This should give me the probability that the curve fits the data. Thank you. – Dave Jarvis Jun 20 '10 at 21:08

The way ANOVA usually works, you have a sum of squares $\sum_i(y_i - \overline{y})^2$ where $\overline{y}$ is the mean $\sum_i y_i /n$ giving the amount of variability in the $y$-values, and you partition it into various (at least two) components, one of which is the sum of squares of residuals, which is the "unexplained" sum of squares. You also have an "explained" sum of squares, and the ratio of the two is often used as a test statistic. Which hypothesis is being tested is more than I want to get into before I ascertain that we're communicating.

You may want to look at this: http://en.wikipedia.org/wiki/Lack-of-fit_sum_of_squares

That fits right into the "ANOVA" context.

One thing you don't get from this is the probability that the model fits. That's something you might get from a Bayesian method, but ANOVA as usually understood is a frequentist method.

-
@Michael: A statistician directly involved with climate data said that I would be best sticking with Spearman's rank correlation, as the data is non-Gaussian. He also mentioned that I could look at other mechanisms to see how well the curve fits, but that that information is relatively meaningless. Thanks for your help! – Dave Jarvis Jul 1 '10 at 5:59