comparing multiple contingency tables, independent data

2010-05-19T03:01:50Z

suppose I have $K$ different methods for forecasting a binary random variable, which I test on independent sets of data, resulting in $K$ contingency tables of values $n_{ijk}$ for $i,j=1,2$ and $k=1,2,...,K$. How can I compare these methods based on the contingency tables? The general case would be nice, but $K=2$ is also very interesting.

I can think of a few approaches:

compare the tetrachoric coefficients of the $K$ tables. this would be especially useful if there were something like Fisher's R-to-Z transform for the tetrachoric coefficient.
compute some statistic on each of the tables, and compare those random variables (I'm not sure if this is a standard problem or not),
something like Goodman's improvement of Stouffer's method, but I cannot access this paper, and was hoping for something a little more recent (more likely to have the latest-greatest, plus computer simulations).

any ideas?

edit: originally had proposed computing 'chi-square' statistic on each table, which is woefully underinformed. the resultant statistic is only distributed as a Chi-square under the null hypothesis of no predictive ability, which I do not want to assume. If there is any test I would like to perform here, it is of the null that the different contingency tables represent the same level of predictive ability.

more information (at request of Kjetil): a number of problems could be posed in this way, I believe. my problem: I have a black box method of forecasting dichotomous y from a vector of input X (if you must know, it is weather-related, but I can say no more...) The method seems to work well when looking at (X,y) pairs from the time period 1970 - 1990, but maybe doesn't work as well on data from the period 1990 to present. I get two contingency tables, one for 1970-1990, the other for 1990-2010. I want to be able to detect whether the forecasting technique has gone bad. As an extension of this problem, you can imagine that the black box was constructed by a third party, and they possibly tuned it around 1990, using all the data available until then. the problem then becomes one of in-sample versus out-of-sample predictive ability. As an extension of this extension, pretend, for the moment, that the third party is withholding the X, and publishes only the predictions of the y, and the actual values of the y (although the latter may be verified through other data streams), thus one cannot even use the X to diagnose the overfitting problems.

another form of the problem might be as follows: given a black box method of forecasting dichotomous y from vector of predictors X, and dichotomous X_0, does the value of X_0 affect the quality of prediction? one could construct 2 contingency tables for the values of X_0, and compare them.

etc.

Answer by Kjetil B Halvorsen for comparing multiple contingency tables, independent data

2010-06-17T02:20:39Z

I think this sort of question should be appropriate for MO, after all, statistics IS (applied) mathematics, and MO is not only pure?

I think the few answers to the question comes from the lack of backgroud, giving applicable answers with background of the applications is DIFFICULT, so the OP should really amend the question.

That being said, i would think about a logistic regression of some kind, maybe with random effects. More details can be given with more deytail in question.

Answer by Tom Smith for comparing multiple contingency tables, independent data

2010-09-17T06:59:47Z

If your prediction functions are linear (or linearisable, or approximably linear) then I think a Chow Test does what you are looking for: it's used in econometrics to test for a structural break in a time series.

comparing multiple contingency tables, independent data - MathOverflow

comparing multiple contingency tables, independent data

Answer by Kjetil B Halvorsen for comparing multiple contingency tables, independent data

Answer by Tom Smith for comparing multiple contingency tables, independent data