I occasionally find that I want to apply a K-S test in the context of unit-testing software that involves random behaviors. Unit testing with sampling statistics is a bit tricky because you want to minimize false-failures.

Extreme value distributions seem like one useful approach, since they make it possible to run a series of experiments, find a maximum D statisic between experimental results and an expected distribution, and measure the probability that D is an outlier using Extreme Value theory.

The idea is that I might run (n) K-S tests, comparing n pairs of samples. This will result in n D-statistic values; the maximum of these values will adhere to some variation of extreme value distribution. I could use the formula for this (or an approximation).

I have never found an extreme value distribution for the K-S D statistic. I suspect it at least adheres to the Weibull form of the EV distribution since its value has a finite maximum, but not even sure of that. I might do some empirical fitting but a more general formula would be even nicer.

UPDATE: Although I have posted an answer that works for a given sample size, it would also be interesting to derive a limiting cdf, that is identify $\alpha$ for the Weibull family of EVD: $$ F(x) = e^{-(-(\frac{x - \mu}{\sigma}))^\alpha} $$

  • 1
    $\begingroup$ The downvote doesn't seem exactly welcoming... Anyway, in case you didn't know it, there's also statistics.stackexchange (= Cross Validated) which might be more welcoming. If other users agree that would be a good move, then we could migrate it over. $\endgroup$ – Todd Trimble Apr 3 '18 at 14:49
  • $\begingroup$ @ToddTrimble Perhaps it is out of scope? Hard for me to tell; I see other questions about properties of KS and/or extreme value distributions in this forum. $\endgroup$ – eje Apr 3 '18 at 17:45

I believe I see the solution. Provided you know the cdf of your iid random variables F(x), the cdf for the maximum value over n iid samples is just (F(x))^n, for example see here.

And I can compute the cdf I need. For a Kolmogorov-Smirnov D statistic over a sample size n (Dn), the quantity sqrt(n)*Dn converges to the Kolmogorov distribution, and its cdf is known; there are also numeric algorithms for computing it, including implementations.

  • 1
    $\begingroup$ If you want a result for finite $n$ (instead of only an asymptotic result), you can use the Dvoretzky–Kiefer–Wolfowitz inequality to get an explicit (non-asymptotic) bound. $\endgroup$ – Bill Bradley Oct 31 '18 at 13:44
  • $\begingroup$ Thanks @BillBradley! That looks very useful! $\endgroup$ – eje Nov 1 '18 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.