2
$\begingroup$

suppose I have $K$ different methods for forecasting a binary random variable, which I test on independent sets of data, resulting in $K$ contingency tables of values $n_{ijk}$ for $i,j=1,2$ and $k=1,2,...,K$. How can I compare these methods based on the contingency tables? The general case would be nice, but $K=2$ is also very interesting.

I can think of a few approaches:

  • compare the tetrachoric coefficients of the $K$ tables. this would be especially useful if there were something like Fisher's R-to-Z transform for the tetrachoric coefficient.
  • compute some statistic on each of the tables, and compare those random variables (I'm not sure if this is a standard problem or not),
  • something like Goodman's improvement of Stouffer's method, but I cannot access this paper, and was hoping for something a little more recent (more likely to have the latest-greatest, plus computer simulations).

any ideas?

edit: originally had proposed computing 'chi-square' statistic on each table, which is woefully underinformed. the resultant statistic is only distributed as a Chi-square under the null hypothesis of no predictive ability, which I do not want to assume. If there is any test I would like to perform here, it is of the null that the different contingency tables represent the same level of predictive ability.

more information (at request of Kjetil): a number of problems could be posed in this way, I believe. my problem: I have a black box method of forecasting dichotomous y from a vector of input X (if you must know, it is weather-related, but I can say no more...) The method seems to work well when looking at (X,y) pairs from the time period 1970 - 1990, but maybe doesn't work as well on data from the period 1990 to present. I get two contingency tables, one for 1970-1990, the other for 1990-2010. I want to be able to detect whether the forecasting technique has gone bad. As an extension of this problem, you can imagine that the black box was constructed by a third party, and they possibly tuned it around 1990, using all the data available until then. the problem then becomes one of in-sample versus out-of-sample predictive ability. As an extension of this extension, pretend, for the moment, that the third party is withholding the X, and publishes only the predictions of the y, and the actual values of the y (although the latter may be verified through other data streams), thus one cannot even use the X to diagnose the overfitting problems.

another form of the problem might be as follows: given a black box method of forecasting dichotomous y from vector of predictors X, and dichotomous X_0, does the value of X_0 affect the quality of prediction? one could construct 2 contingency tables for the values of X_0, and compare them.

etc.

Improve this question
$\endgroup$
1
  • $\begingroup$ We don't seem to have very many people here for answering this sort of question. On the whole questions like this are more interesting and relevant to scientists, not mathematicians. I think someone should start a new website for these sorts of questions. $\endgroup$
    – Noah Snyder
    Commented May 20, 2010 at 0:35

2 Answers 2

Reset to default
2
$\begingroup$

If your prediction functions are linear (or linearisable, or approximably linear) then I think a Chow Test does what you are looking for: it's used in econometrics to test for a structural break in a time series.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ As a fellow 'meteorologist', needless to say, I think the question is quite interesting. Two comments- 1)perhaps if it was phrased as a financial mathematics question, the intuition might be clearer and 2) while weather forecasting tends to be quite proprietary, I have wondered if methods of algebraic statistics might not be useful for this sort of question. As a bonus, if this was so, it would tie the question into some very interesting main-stream mathematics. $\endgroup$
    – meh
    Commented Dec 25, 2010 at 2:59
0
$\begingroup$

I think this sort of question should be appropriate for MO, after all, statistics IS (applied) mathematics, and MO is not only pure?

I think the few answers to the question comes from the lack of backgroud, giving applicable answers with background of the applications is DIFFICULT, so the OP should really amend the question.

That being said, i would think about a logistic regression of some kind, maybe with random effects. More details can be given with more deytail in question.

Improve this answer
$\endgroup$
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .