Welcome to MathOverflow

Question

0

If i test one truly alternative hypothesis with 95% power, and one truly null hypothesis, what is the probability that the the p value from the truly alternative hypothesis test will be less than the p value from the truly null hypothesis test?

If i test two hypotheses, one of which has a null that is -in fact- false and the other has a null that is -in fact- true, I want to know the probability that the first will test will obtain a p value less than that second.

I can simulate the situation in R, and come up with a reasonable estimate that way, but I want to know how I can impute the answer exactly.

Using the program R:

a=vector()
b=vector()
for (i in 1:1000) {
+ ai = rnorm(108, mean = 4, sd = 20)
+ a = c(a,t.test(ai)p.value)
+ bi = rnorm(108)
+ b = c(b,t.test(bi)$p.value)
+ }
q=rep(0,length=length(a))
q[a less than b]=1
mean(q)
[1] 0.844

So, I find that approximately 85% of the time, the truly alternative hypothesis yields a lower p value, but how can I impute this result. I want to prove it mathematically.

Thanks for your help,

Nick

edited May 18 2011 at 7:04

asked May 17 2011 at 18:18

Nick
1●1

1

Usually one speaks of (1) a probability given that the null hypothesis is true, or (2) the largest possible probability given that the null hypothesis is true, if there is more than one hypothetical population distribution consistent with the null hypothesis, or of (3) the probability given a particular distribution consistent with the alternative hypothesis. All that assumes a frequentist setting. In a Bayesian setting, one can assign probabilities to the hypotheses themselves, but then one would not normally be concerned with p-values, but rather with posterior probabilities. – Michael Hardy May 17 2011 at 19:04

What's the difference between a "truly" [alternative/null] hypothesis and a (ordinary meaning) [alternative/null] hypothesis? – Ricky Demer May 17 2011 at 19:41

By truly alternative, I mean that the null hypothesis is in fact false. By truly null, I mean that the null hypothesis is in fact true. I'm asking: if I have 95% power to reject one null hypothesis that is in fact false, and I also test another null hypothesis that is in fact true (power doesn't apply to this hypothesis, because a type II error is impossible), what is the probability that the p value obtained for the first hypothesis is less than the p value obtained for the second? – Nick May 17 2011 at 19:54

Also, I should add that I am not interested in whether I call one, or both, or neither p values significant, I'm only interested in the probability that one is less than the other. – Nick May 17 2011 at 19:56

1

stats.stackexchange.com may get you more interest (and answers) for this question. – David Roberts May 18 2011 at 1:52

Probability that X<=Y when X and Y are p values from two different hypotheses

Remember to vote up questions/answers you find interesting or helpful (requires 15 reputation points)

You can accept an answer to one of your own questions by clicking the check mark next to it. This awards 15 reputation points to the person who answered and 2 reputation points to you.