# Distribution for probability of an incorrect inference based on a comparison of only two samples?

I'm trying to demonstrate the problems of how taking a sample and assuming it reflects the population accurately can be problematic.

Imagine say an urn with some large number of balls, black and white, with white P and black Q probabilities. I know the true parameters, the sampler does not.

The sampler draws some N balls with replacement, then infers P is the actual population parameter based only on that fraction of whites vs blacks drawn. They later do a second sampling of the same number of balls, and note a change that is "significant" (and assert perhaps someone changed the contents).

I've been using a simulation that simply spews pairs of binomial trials, tallies the successes for the first of each, then uses that inferred P to determine if the second member of the pair would be seen as significantly different by the sampler, then test if in fact the second of the pair is not an out-lier based on the true population P, finally counting those that would have been incorrectly marked significant when they are within the bounds of insignificance using the true P.

Is there a distribution, that in the fashion of Skellam for probabilities of differences of means, gives the probability of drawing an incorrect conclusion when comparing the outcomes of two samples? That is, one that gives the probability of incorrectly attributing significance to an outcome based on the results of the prior outcome, when the actual population parameters are known (but not, as said, to the sampler)?

Thx.

-
Hmmm... Isn't that just a fairly straightforward exercise in definitions and CLT? I mean, what is the difficulty you have with it? Am I overlooking some subtlety? – fedja May 25 '13 at 12:35
@fedja: It doesn't look straightforward to me since the samples are discrete and you are estimating not just the mean, but also the variance from the first sample. – Douglas Zare May 25 '13 at 23:11
You mean you want to go well beyond the mere asymptotic for large $N$? When $N$ is large and the confidence level is not ridiculously small, we have the estimated variance equal to $1+o(1)$ times the true one in the observation range that matters at all plus, even when the observed $p$ deviates from $P$ by a noticeable amount, we have essentially the same probability to increase and to decrease the variance, so the effect should be of order $O(|p-P|^2)$ rather than mere $O(|p-P|)$. So, unless some extreme regimes are of importance, we can just use the normal approximation with fixed variance. – fedja May 26 '13 at 14:31