So I'm a self-learner which is always dangerous because I don't have anything to test if I am understanding things correctly, so I wanted to ask what is wrong/right with my assumptions.
When reading about this stuff this is what I implied. Unless the null is true, by increasing sample sizes you will decrease the p value. So it seems to me that the null will never be 100% accurate (i.e. If my null is that the avg weight of men is 70kg, in reality it will never be exactly 70, if it is 70.000001 in reality I would eventually detect this as significant my sample gets huge).
So if that logic is correct then as my sample size goes to infinity, unless my null is accurate to infinity or my p value is infinitely small, I will always detect a difference in a two tailed p test. This seems to take meaning away from p because now how do I know if there is a difference that matters in "real life" or if I just took too big of a sample and forced a meaningless p value??
If that is true, shouldn't we be using some kind of measure that correlates this: if I detect a small p with a small sample it is more likely to have a large difference? That way somehow we could have some idea of the actual difference and if it is actually important in real life. Is this where confidence intervals come in play? And if so how does that work and why aren't CI often reported in publications?
I'm going crazy trying to figure this out... Also, I know that a one-tailed p value is at least useful for detecting the direction of the real average, but it seems to have a similar problem. If I get a small p for the question "is the avg weight of men >70kg?". If the weight is 70.001kg and not 70kg it really doesn't matter in real life. But how could I tell if in real life it is 70.0001 or 80?