1
$\begingroup$

I have one continuous variable, Variable, and two categorical variables, Factor1 and Factor2, each comprising two levels. What does it mean if

1) According to a t-test, the difference in Variable between the two levels in Factor2 are statistically significant only for a subset of Factor1 (only within one of its levels).

2) The pairwise comparisons according to Tukey's Honest Significant Difference test and the F-statistic for each of the categorical variables in a 2-way ANOVA both show that the influence of these categorical variables are not significant (and interactions also not significant).

?

This seems like an elementary textbook question but my lack of formal statistical training is beginning to show...

Thank you in advance for your responses.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

It is very often the case that some subset of your data will come out to be statistically significant by random chance. If you are running t-tests among the levels given the level of each other factor, that's four tests. Your chance of one of those four comparisons being significant at $\alpha=0.05$ is $1 - (0.95)^4$, or about 20%.

The ANOVA, F-test, and HSD all account for this kind of multiple comparison. It's what they were designed to handle.

It's actually quite a nice example of why multiple comparisons are important to take into account.

Improve this answer
$\endgroup$
3
  • $\begingroup$ It is not a good idea to say "be statistically significant by random chance" even if we can understand the spirit. The sentence "Your chance of one of those four comparisons being significant at =005 is $1−(095)^4$, or about $20\%$" is imprecise: is it the chance to be declared significant truly ? falsely ? anyway, I don't see the connection with the question which seems more related to the difference between two way for accounting for effects? Am I wrong ? $\endgroup$
    – robin girard
    Commented Jul 15, 2010 at 6:09
  • $\begingroup$ The phrase "statistically significant by random chance" seems to be standard, correct, concise and easily understood. Why is it not a good idea to use that phrase? $\endgroup$
    – T..
    Commented Jul 27, 2010 at 18:00
  • $\begingroup$ It is the chance to be declared significant falsely, otherwise we would be worried about $\beta$, not $\alpha$. Neyman-Pearson hypothesis testing is based on fixing $\alpha$ and letting everything else fall where it will. From a decision theory point of view, it is absolutely correct to say "statistically significant by random chance." For decisions D0 and D1 corresponding to hypotheses H0 and H1, we acquire some data, and apply some procedure to get D0 or D1. There is a probability of getting D1 when H0 is true. $\endgroup$
    – madhadron
    Commented Aug 11, 2010 at 14:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .