Interpretations among t-test, ANOVA, Tukey HSD results? - MathOverflow

Interpretations among t-test, ANOVA, Tukey HSD results?

2010-05-03T09:44:36Z

I have one continuous variable, Variable, and two categorical variables, Factor1 and Factor2, each comprising two levels. What does it mean if

1) According to a t-test, the difference in Variable between the two levels in Factor2 are statistically significant only for a subset of Factor1 (only within one of its levels).

2) The pairwise comparisons according to Tukey's Honest Significant Difference test and the F-statistic for each of the categorical variables in a 2-way ANOVA both show that the influence of these categorical variables are not significant (and interactions also not significant).

This seems like an elementary textbook question but my lack of formal statistical training is beginning to show...

Thank you in advance for your responses.

Answer by madhadron for Interpretations among t-test, ANOVA, Tukey HSD results?

2010-07-13T14:21:19Z

It is very often the case that some subset of your data will come out to be statistically significant by random chance. If you are running t-tests among the levels given the level of each other factor, that's four tests. Your chance of one of those four comparisons being significant at $\alpha=0.05$ is $1 - (0.95)^4$, or about 20%.

The ANOVA, F-test, and HSD all account for this kind of multiple comparison. It's what they were designed to handle.

It's actually quite a nice example of why multiple comparisons are important to take into account.

Answer by PeterR for Interpretations among t-test, ANOVA, Tukey HSD results?

2010-07-27T16:36:06Z

You might want to post this question at http://stats.stackexchange.com/ Went public beta yesterday