Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

If i test one truly alternative hypothesis with 95% power, and one truly null hypothesis, what is the probability that the the p value from the truly alternative hypothesis test will be less than the p value from the truly null hypothesis test?

If i test two hypotheses, one of which has a null that is -in fact- false and the other has a null that is -in fact- true, I want to know the probability that the first will test will obtain a p value less than that second.

I can simulate the situation in R, and come up with a reasonable estimate that way, but I want to know how I can impute the answer exactly.

Using the program R:

a=vector()
b=vector()
for (i in 1:1000) {
+ ai = rnorm(108, mean = 4, sd = 20)
+ a = c(a,t.test(ai)p.value)
+ bi = rnorm(108)
+ b = c(b,t.test(bi)$p.value)
+ }
q=rep(0,length=length(a))
q[a less than b]=1
mean(q)
[1] 0.844

So, I find that approximately 85% of the time, the truly alternative hypothesis yields a lower p value, but how can I impute this result. I want to prove it mathematically.

Thanks for your help,

Nick

share|improve this question
1  
Usually one speaks of (1) a probability given that the null hypothesis is true, or (2) the largest possible probability given that the null hypothesis is true, if there is more than one hypothetical population distribution consistent with the null hypothesis, or of (3) the probability given a particular distribution consistent with the alternative hypothesis. All that assumes a frequentist setting. In a Bayesian setting, one can assign probabilities to the hypotheses themselves, but then one would not normally be concerned with p-values, but rather with posterior probabilities. –  Michael Hardy May 17 '11 at 19:04
    
What's the difference between a "truly" [alternative/null] hypothesis and a (ordinary meaning) [alternative/null] hypothesis? –  Ricky Demer May 17 '11 at 19:41
    
By truly alternative, I mean that the null hypothesis is in fact false. By truly null, I mean that the null hypothesis is in fact true. I'm asking: if I have 95% power to reject one null hypothesis that is in fact false, and I also test another null hypothesis that is in fact true (power doesn't apply to this hypothesis, because a type II error is impossible), what is the probability that the p value obtained for the first hypothesis is less than the p value obtained for the second? –  Nick May 17 '11 at 19:54
    
Also, I should add that I am not interested in whether I call one, or both, or neither p values significant, I'm only interested in the probability that one is less than the other. –  Nick May 17 '11 at 19:56
1  
stats.stackexchange.com may get you more interest (and answers) for this question. –  David Roberts May 18 '11 at 1:52
add comment

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.