1
$\begingroup$

(Sorry if this is a noob question. I'm a mathematician learning statistics.)

I would like to know if it's sound (or advisable) to test many p-values against the continuous uniform distribution using Kolmogorov-Smirnov or Anderson-Darling.

For example, suppose I want to demonstrate that Rock-Paper-Scissors is a fair game. My H0 is that all players win 1/3 of their games. I get 100 people to play 100 others (varying number of games for each pairing, varying number of total games for each player).

For each player, I can compare their wins against their expected wins (1/3) with chi-squared. This would give me 100 p-values. With so many players, it's probably that a few will not meet my significance level. So I figure I can test the p-values themselves using Anderson-Darling. I would only reject H0 if this last test failed.

Any thoughts are appreciated, thanks.

Improve this question
$\endgroup$
2
  • $\begingroup$ I guess the basic idea is sound, but there are problems: -in your example the chisquared distribution is only asymptotic, that is, approximate. Your test could simply detect the deviation from that approximation! You should look into multiple hypothesis testing, for example en.wikipedia.org/wiki/False_discovery_rate and the references therein. $\endgroup$
    – kjetil b halvorsen
    Apr 4, 2014 at 10:51
  • $\begingroup$ Thanks @KjetilBHalvorsen. I think FDR will be very helpful. That said, I think I've convinced myself that testing p-values is OK, assuming the p-values came from a "correct" test. In this case, by correct I mean that over all inputs, the test produces p-values that are uniformly distributed. With a random input, the test effectively becomes a random number generator on [0,1]. This should qualify it for the AD test. (Of course, taking your point that the chi-squared test in particular is approximate). $\endgroup$
    – user41416
    Apr 4, 2014 at 12:45

1 Answer 1

Reset to default
1
$\begingroup$

I think what you're looking for is the higher criticism statistic, which was suggested originally by Tukey: he compares the number of $\alpha$-level significant detections to the expected number of such detections you should expect under the joint null. Higher criticism has some good mathematical properties, as shown by Donoho and Jin (http://www.jstor.org/stable/3448581).

In practice, however, I'd probably follow Kjetil's advice and just do FDR control.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.