MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have one continuous variable, Variable, and two categorical variables, Factor1 and Factor2, each comprising two levels. What does it mean if

1) According to a t-test, the difference in Variable between the two levels in Factor2 are statistically significant only for a subset of Factor1 (only within one of its levels).

2) The pairwise comparisons according to Tukey's Honest Significant Difference test and the F-statistic for each of the categorical variables in a 2-way ANOVA both show that the influence of these categorical variables are not significant (and interactions also not significant).


This seems like an elementary textbook question but my lack of formal statistical training is beginning to show...

Thank you in advance for your responses.

share|cite|improve this question
up vote 2 down vote accepted

It is very often the case that some subset of your data will come out to be statistically significant by random chance. If you are running t-tests among the levels given the level of each other factor, that's four tests. Your chance of one of those four comparisons being significant at $\alpha=0.05$ is $1 - (0.95)^4$, or about 20%.

The ANOVA, F-test, and HSD all account for this kind of multiple comparison. It's what they were designed to handle.

It's actually quite a nice example of why multiple comparisons are important to take into account.

share|cite|improve this answer
It is not a good idea to say "be statistically significant by random chance" even if we can understand the spirit. The sentence "Your chance of one of those four comparisons being significant at =005 is $1−(095)^4$, or about $20\%$" is imprecise: is it the chance to be declared significant truly ? falsely ? anyway, I don't see the connection with the question which seems more related to the difference between two way for accounting for effects? Am I wrong ? – robin girard Jul 15 '10 at 6:09
The phrase "statistically significant by random chance" seems to be standard, correct, concise and easily understood. Why is it not a good idea to use that phrase? – T.. Jul 27 '10 at 18:00
It is the chance to be declared significant falsely, otherwise we would be worried about $\beta$, not $\alpha$. Neyman-Pearson hypothesis testing is based on fixing $\alpha$ and letting everything else fall where it will. From a decision theory point of view, it is absolutely correct to say "statistically significant by random chance." For decisions D0 and D1 corresponding to hypotheses H0 and H1, we acquire some data, and apply some procedure to get D0 or D1. There is a probability of getting D1 when H0 is true. – madhadron Aug 11 '10 at 14:51

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.