more information (at request of Kjetil): a number of problems could be posed in this way, I believe. my problem:I have a black box method of forecasting dichotomous y from a vector of input X (if you must know, it is weather-related, but I can say no more...) The method seems to work well when looking at (X,y) pairs from the time period 1970 - 1990, but maybe doesn't work as well on data from the period 1990 to present. I get two contingency tables, one for 1970-1990, the other for 1990-2010. I want to be able to detect whether the forecasting technique has gone bad.As an extension of this problem, you can imagine that the black box was constructed by a third party, and they possibly tuned it around 1990, using all the data available until then. the problem then becomes one of in-sample versus out-of-sample predictive ability.As an extension of this extension, pretend, for the moment, that the third party is withholding the X, and publishes only the predictions of the y, and the actual values of the y (although the latter may be verified through other data streams), thus one cannot even use the X to diagnose the overfitting problems.

another form of the problem might be as follows: given a black box method of forecasting dichotomous y from vector of predictors X, and dichotomous X_0, does the value of X_0 affect the quality of prediction? one could construct 2 contingency tables for the values of X_0, and compare them.

etc.

2 edit out reference to chi-square, which was not right.

suppose I have $K$ different methods for forecasting a binary random variable, which I test on independent sets of data, resulting in $K$ contingency tables of values $n_{ijk}$ for $i,j=1,2$ and $k=1,2,...,K$. How can I compare these methods based on the contingency tables? The general case would be nice, but $K=2$ is also very interesting.

I can think of a few approaches:

• compare the tetrachoric coefficients of the $K$ tables. this would be especially useful if there were something like Fisher's R-to-Z transform for the tetrachoric coefficient.
• compute the Chi-square some statistic on each of the tables, and compare those random variables (I'm not sure if this is a standard problem or not),
• something like Goodman's improvement of Stouffer's method, but I cannot access this paper, and was hoping for something a little more recent (more likely to have the latest-greatest, plus computer simulations).

any ideas?

edit: originally had proposed computing 'chi-square' statistic on each table, which is woefully underinformed. the resultant statistic is only distributed as a Chi-square under the null hypothesis of no predictive ability, which I do not want to assume. If there is any test I would like to perform here, it is of the null that the different contingency tables represent the same level of predictive ability.

1

# comparing multiple contingency tables, independent data

suppose I have $K$ different methods for forecasting a binary random variable, which I test on independent sets of data, resulting in $K$ contingency tables of values $n_{ijk}$ for $i,j=1,2$ and $k=1,2,...,K$. How can I compare these methods based on the contingency tables? The general case would be nice, but $K=2$ is also very interesting.

I can think of a few approaches:

• compare the tetrachoric coefficients of the $K$ tables. this would be especially useful if there were something like Fisher's R-to-Z transform for the tetrachoric coefficient.
• compute the Chi-square statistic on each of the tables, and compare those random variables (I'm not sure if this is a standard problem or not),
• something like Goodman's improvement of Stouffer's method, but I cannot access this paper, and was hoping for something a little more recent (more likely to have the latest-greatest, plus computer simulations).

any ideas?