I'm trying to demonstrate the problems of how taking a sample and assuming it reflects the population accurately can be problematic.
Imagine say an urn with some large number of balls, black and white, with white P and black Q probabilities. I know the true parameters, the sampler does not.
The sampler draws some N balls with replacement, then infers P is the actual population parameter based only on that fraction of whites vs blacks drawn. They later do a second sampling of the same number of balls, and note a change that is "significant" (and assert perhaps someone changed the contents).
I've been using a simulation that simply spews pairs of binomial trials, tallies the successes for the first of each, then uses that inferred P to determine if the second member of the pair would be seen as significantly different by the sampler, then test if in fact the second of the pair is not an out-lier based on the true population P, finally counting those that would have been incorrectly marked significant when they are within the bounds of insignificance using the true P.
Is there a distribution, that in the fashion of Skellam for probabilities of differences of means, gives the probability of drawing an incorrect conclusion when comparing the outcomes of two samples? That is, one that gives the probability of incorrectly attributing significance to an outcome based on the results of the prior outcome, when the actual population parameters are known (but not, as said, to the sampler)?