Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

So I'm a self-learner which is always dangerous because I don't have anything to test if I am understanding things correctly, so I wanted to ask what is wrong/right with my assumptions.

When reading about this stuff this is what I implied. Unless the null is true, by increasing sample sizes you will decrease the p value. So it seems to me that the null will never be 100% accurate (i.e. If my null is that the avg weight of men is 70kg, in reality it will never be exactly 70, if it is 70.000001 in reality I would eventually detect this as significant my sample gets huge).

So if that logic is correct then as my sample size goes to infinity, unless my null is accurate to infinity or my p value is infinitely small, I will always detect a difference in a two tailed p test. This seems to take meaning away from p because now how do I know if there is a difference that matters in "real life" or if I just took too big of a sample and forced a meaningless p value??

If that is true, shouldn't we be using some kind of measure that correlates this: if I detect a small p with a small sample it is more likely to have a large difference? That way somehow we could have some idea of the actual difference and if it is actually important in real life. Is this where confidence intervals come in play? And if so how does that work and why aren't CI often reported in publications?

I'm going crazy trying to figure this out... Also, I know that a one-tailed p value is at least useful for detecting the direction of the real average, but it seems to have a similar problem. If I get a small p for the question "is the avg weight of men >70kg?". If the weight is 70.001kg and not 70kg it really doesn't matter in real life. But how could I tell if in real life it is 70.0001 or 80?

share|improve this question

closed as off-topic by Will Jagy, Douglas Zare, Suvrit, Jeremy Rouse, S. Carnahan Aug 10 at 4:05

This question appears to be off-topic. The users who voted to close gave these specific reasons:

  • "MathOverflow is for mathematicians to ask each other questions about their research. See Math.StackExchange to ask general questions in mathematics." – Will Jagy, S. Carnahan
  • "This question does not appear to be about research level mathematics within the scope defined in the help center." – Douglas Zare, Suvrit, Jeremy Rouse
If this question can be reworded to fit the rules in the help center, please edit the question.

2  
I think stats.stackexchange.com might be more suitable for this question –  Yemon Choi Aug 9 at 23:39
    
Crossposted on reddit: reddit.com/r/math/comments/2d3tgt/… –  Nathaniel Johnston Aug 10 at 3:57

1 Answer 1

Everything you say is correct. If you want a useful search term, people often talk about the "effect size" as as well as the p-value. It is entirely possible to have a small p-value but an effect size which is of no practical use, as in your example.

To answer your questions:

This seems to take meaning away from p because now how do I know if there is a difference that matters in "real life" or if I just took too big of a sample and forced a meaningless p value??

The situation is actually even worse than this because the p-value is always founded on "assumptions" and a small p-value can just as easily be interpreted as evidence against all of these assumptions together. But usually some of the assumptions are waved away by saying, for example, "the plot looks normal, so it's ok to use the X test."

As far as I understand, even very early users of p-values understood that they might be useless in large samples. But people have gradually forgotten this over the decades.

Is this where confidence intervals come in play? And if so how does that work and why aren't CI often reported in publications?

Yes, confidence intervals are one way of measuring effect size. They are often reported in publications in some areas, but not others. The reason why they are often not reported is because the standard of statistics is really really low. Some statisticians have recently been encouraging psychologists to use confidence intervals. This is called "the New Statistics" (google it.)

But how could I tell if in real life it is 70.0001 or 80?

A confidence interval would be one way of doing this. A credible interval, if you learn Bayesian statistics, is another way. If you were just given the p-value by itself, you have literally no way of knowing. p-values just aren't very useful.

share|improve this answer
    
Thanks a lot for this response, I really want to find a better way. In medical research it seems we rarely incorporate a statistician (at least in my experience) into the analysis, figuring that we can get a program to do it for us. The interpretation of these tests we run really should be run by a statistician in order to make sure inappropriate inferences aren't made. Also sorry for posting this is the wrong section, I am asking for the sake of some of my research but I should have read the rules better. Thanks for the response! –  Synchro_tr0n Aug 12 at 4:08

Not the answer you're looking for? Browse other questions tagged or ask your own question.