4
$\begingroup$

I have conducted a within-subject user study. Each user works with 3 user interfaces and reads a couple of tweets in each UI and we record how many milliseconds they spend on reading each tweet/comment for each UI

I was doing ANOVA to see if the type of UI contributes to the amount of time people spend on each UI. Everything was shiny until I noticed that the dataset fails on Bartlett's test with a very low p-value. I used an "ANOVA on Ranks" instead (ranked the timing values and ran ANOVA again) of ANOVA there and everything became shiny again!

My question is:

  • Can I use "ANOVA on Ranks" when there is no homogeneity of variances. Do I need to do a transformation? or use a Friedman test instead?
  • The more important question is what should I use as the follow up method (since I cannot use any t-test type test)

Thanks a lot for the help

Cheers

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

ANOVA on ranks in principle needs homogeneity of variances, but in practice can be quite robust without. I once did some simple fast simulations in R and was astonished to see how little the non-constant variances mattered. You could do the same. Try transformations! Plot the data. If the effect is large it might be quite obvious after looking at the plot. Or better, post the data here in some easy-readable format and I have a look.

Improve this answer
$\endgroup$
1
$\begingroup$

I think you can still use a t-test, but you have to keep in mind that what you call "an effect" can also appear in the variance. As a consequence, I think you should add a test to your t-test to measure difference in the variances.

Simple example, consider the case when $X$ is a $\mathcal{N}(0,1)$ and $Y$ is a $\mathcal{N}(\mu,\sigma)$ and you want to say if $Y$ is the result of "an effect" or it is not.

You can look at a t-test like that: $\frac{|\bar{X}-\bar{Y}|}{\sqrt{var(X)+var(Y)}}$. It works fine even with different variances but it will only check for differences in the mean (i.e $\mu\neq 1$). If you want to test something on the variances also, you will have to test simultaneously (take care of the level of your test) the equality of the mean and the equality of the variances.

More General setting If an effect can appear in a more complex way, you may be obliged to concider a non parametric godness of fit testing procedure to which belong (I guess) the rank test.

Hop this help !

Improve this answer
$\endgroup$
0
$\begingroup$

Have you tried taking the logarithm of the times and then test for homogeneity? It has been my experience that when dealing with positive responses that were task related that the log transformation often does the trick.

A more robust solution to your problem is to use the Box-Cox transformation, a generalization of the above transform with a parameter lambda, that when fitted and found to be 1/2, turns out to be the log transform.

One more issue, is this a factorial design or a nested design? The answer to the question depends on how you ran the experiment.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .