ANOVA analysis with no homogeneity of variances

I have conducted a within-subject user study. Each user works with 3 user interfaces and reads a couple of tweets in each UI and we record how many milliseconds they spend on reading each tweet/comment for each UI

I was doing ANOVA to see if the type of UI contributes to the amount of time people spend on each UI. Everything was shiny until I noticed that the dataset fails on Bartlett's test with a very low p-value. I used an "ANOVA on Ranks" instead (ranked the timing values and ran ANOVA again) of ANOVA there and everything became shiny again!

My question is:

• Can I use "ANOVA on Ranks" when there is no homogeneity of variances. Do I need to do a transformation? or use a Friedman test instead?
• The more important question is what should I use as the follow up method (since I cannot use any t-test type test)

Thanks a lot for the help

Cheers

-

ANOVA on ranks in principle needs homogeneity of variances, but in practice can be quite robust without. I once did some simple fast simulations in R and was astonished to see how little the non-constant variances mattered. You could do the same. Try transformations! Plot the data. If the effect is large it might be quite obvious after looking at the plot. Or better, post the data here in some easy-readable format and I have a look.

-

I think you can still use a t-test, but you have to keep in mind that what you call "an effect" can also appear in the variance. As a consequence, I think you should add a test to your t-test to measure difference in the variances.

Simple example, consider the case when $X$ is a $\mathcal{N}(0,1)$ and $Y$ is a $\mathcal{N}(\mu,\sigma)$ and you want to say if $Y$ is the result of "an effect" or it is not.

You can look at a t-test like that: $\frac{|\bar{X}-\bar{Y}|}{\sqrt{var(X)+var(Y)}}$. It works fine even with different variances but it will only check for differences in the mean (i.e $\mu\neq 1$). If you want to test something on the variances also, you will have to test simultaneously (take care of the level of your test) the equality of the mean and the equality of the variances.

More General setting If an effect can appear in a more complex way, you may be obliged to concider a non parametric godness of fit testing procedure to which belong (I guess) the rank test.

Hop this help !

-