# Interpretations among t-test, ANOVA, Tukey HSD results?

I have one continuous variable, Variable, and two categorical variables, Factor1 and Factor2, each comprising two levels. What does it mean if

1) According to a t-test, the difference in Variable between the two levels in Factor2 are statistically significant only for a subset of Factor1 (only within one of its levels).

2) The pairwise comparisons according to Tukey's Honest Significant Difference test and the F-statistic for each of the categorical variables in a 2-way ANOVA both show that the influence of these categorical variables are not significant (and interactions also not significant).

?

This seems like an elementary textbook question but my lack of formal statistical training is beginning to show...

-

It is very often the case that some subset of your data will come out to be statistically significant by random chance. If you are running t-tests among the levels given the level of each other factor, that's four tests. Your chance of one of those four comparisons being significant at $\alpha=0.05$ is $1 - (0.95)^4$, or about 20%.
It is not a good idea to say "be statistically significant by random chance" even if we can understand the spirit. The sentence "Your chance of one of those four comparisons being significant at =005 is $1−(095)^4$, or about $20\%$" is imprecise: is it the chance to be declared significant truly ? falsely ? anyway, I don't see the connection with the question which seems more related to the difference between two way for accounting for effects? Am I wrong ? – robin girard Jul 15 '10 at 6:09
It is the chance to be declared significant falsely, otherwise we would be worried about $\beta$, not $\alpha$. Neyman-Pearson hypothesis testing is based on fixing $\alpha$ and letting everything else fall where it will. From a decision theory point of view, it is absolutely correct to say "statistically significant by random chance." For decisions D0 and D1 corresponding to hypotheses H0 and H1, we acquire some data, and apply some procedure to get D0 or D1. There is a probability of getting D1 when H0 is true. – madhadron Aug 11 '10 at 14:51