more information (at request of Kjetil): a number of problems could be posed in this way, I believe. my problem:I have a black box method of forecasting dichotomous y from a vector of input X (if you must know, it is weather-related, but I can say no more...) The method seems to work well when looking at (X,y) pairs from the time period 1970 - 1990, but maybe doesn't work as well on data from the period 1990 to present. I get two contingency tables, one for 1970-1990, the other for 1990-2010. I want to be able to detect whether the forecasting technique has gone bad.As an extension of this problem, you can imagine that the black box was constructed by a third party, and they possibly tuned it around 1990, using all the data available until then. the problem then becomes one of in-sample versus out-of-sample predictive ability.As an extension of this extension, pretend, for the moment, that the third party is withholding the X, and publishes only the predictions of the y, and the actual values of the y (although the latter may be verified through other data streams), thus one cannot even use the X to diagnose the overfitting problems.
another form of the problem might be as follows: given a black box method of forecasting dichotomous y from vector of predictors X, and dichotomous X_0, does the value of X_0 affect the quality of prediction? one could construct 2 contingency tables for the values of X_0, and compare them.