# Is there a general theory supporting the construction of conditional confidence intervals?

Conditional confidence intervals are intervals whose confidence statements apply even after considering the actual data collected (i.e., conditional on the data actually observed, not averaged over all possible samples). Note: this is a well-defined frequentist concept, not an attempt to invest parameters with randomness.

The key idea that allows the repeated-sampling interpretation is the concept of a relevant subset -- conditional inference aims to restrict repeated sampling claims (e.g., coverage) to samples from only this specific subset of the sample space. Please refer to a related question here.

I am familiar with some specific methods for doing this, including the famous paper by Groutis and Casella for the case of the t-distribution and earlier work for the Cauchy an Logistic distributions. There is also work in this area for the double exponential. However, all these seem rather ad-hoc, with their dependence on just the right ancillary statistic or some clever observation of invariance, etc.

I'm hoping that there has emerged some general theoretical approach to forming "correct" conditional inferences, in much the same way that Bayesians know how to construct a "correct" Bayesian inference.

The best exposition of what appears to be such a theory is by (again!) Casella in a brief paper. But it is only a sketch and only suggests possible approaches.

Can someone knowledgeable in modern frequentist conditional inference describe the current methods for forming frequentist conditional confidence sets (i.e., sets that make post-data confidence guarantees [basically by taking into account the relevant subsets selected by the sample at hand])?

Ok, so I did some more digging and it turns out that Casella and Groutis (yes, again!) wrote a nice paper on approaches to frequentist post-data inferences. I won't summarize the 35 page paper here, but I'll point out the key points for purposes of making my answer self-contained.

First, they convincingly demonstrate that post-data confidence assignments cannot be derived using the usual pre-data covering metrics nor by partitioning the sample space. The former is relatively obvious, but the latter goes to the heart of Fisher's admonition on conditional inference via ancillaries -- Casella and Groutis argue that one cannot to rely on ancillary statistics to partition the sample space, since these are not guaranteed to exist and even if they do they can be non-unique (they cite well-known results from Basu here).

Second, they critique an early approach by Kiefer to post-data frequentist inference that relied on the experimenter partitioning the sample space. Simply put, one cannot get around the "reference class problem" and so there is no objective basis on which to perform this partitioning.

Unfortunately, if you don't create some sort of coarse partition (i.e., containing more than just the observed sample), then you are unable to develop a suitable reference set for coverage calculations (you're basically left evaluating the apriori likelihood of each parameter...quite a non-frequentist concept). Bayesians and Likelihoodists address this by inferring directly from the likelihood (with prior for Bayesians) at the expense of repeated-sampling guarantees on error rates.

## Post-Data Confidence as an Estimator

The "big idea" Casella and Groutis discuss is that freqentist post-data confidence should be seen as an estimation problem, not a probability calculation problem (i.e., calculating pre-data coverage probability). They make this transition by defining "confidence" as an estimate of the mean value of an indicator function for the non-random interval actually observed (I'll reproduce their math here because it is the heart of the approach):

$$P_{\theta}(\theta \in C(X)|X=x) = P_{\theta}(\theta \in C(x)) = I(\theta \in C(x))$$

The development of frequentist post-data inference follows from above using two obvious mathematical ideas:

1. A fixed interval either does or does not contain a given number (e.g., $[2,3]$ does not contain $5$).
2. We can replace probabilities with expectations when we use indicator functions over a set.

They then discuss how one should evaluate post-data confidence using a confidence estimator $\gamma(X)$ which can then be assessed using a loss function (just like any other estimator). They showed this for the squared-error loss:

$$R(\theta,\gamma(X)) = E_{\theta}\left[I\left(\theta \in C(X)\right) - \gamma(X)\right]^2$$

Using this approach, we can then evaluate various confidence estimators to determine (a) their overall accuracy and (b) which one to use.

For example, if $R(\theta,\gamma_1) \leq R(\theta,\gamma_2) \; \forall \theta$ then we would want to use the estimates provided by $\gamma_1$ as a post-data measure of confidence (I'd add that we should probably want $RSME(\gamma) < \epsilon$ for some small $\epsilon$ like $0.05$ ... but this is a theory paper so that's probably not its focus).

Following the above technique, a post-data confidence assessment would then read like this: "$C(x)$ contains the true parameter with confidence $\gamma(x)$", where it is understood that "confidence" is an estimate of the expected value of the indicator.

Before data is collected, the expected value of $\gamma(X)$ provides a pre-data measure of confidence in the procedure defined by $\langle C(X),\gamma(X)\rangle$. Hence $E[\gamma(X)]$ resembles a coverage probability, so we can set our pre-data confidence using the properties of $\gamma(X)$ to get pre-data estimate of coverage as well (i.e., the usual theory).

## Discussion

While I find this a theoretically satisfying result (which is why I'm posting this as an answer to my question), I find it rather useless in practice. Casella and Groutis hint as much by way of a single sentence on page 19:

"Construction of such estimators $\mathrm{[\gamma(X)]}$ is sometimes technically involved."

To say that this is an understatement would in itself be an understatement. Even for well-known, analytically "nice" distributions like the normal distribution, the underlying mathematics is rather intense. I can't imagine deriving estimators for mixed-effects models and those with nuisance parameters in an analytically tractable way.

However, non-parametric/machine-learning approaches to developing estimators via simulations across broad grid of parameter values may yield interesting shortcuts for more realistic scenarios...but I'm not holding my breath. I think I'll stick with Bayesian inference with posterior model checking and frequentist evaluations.