I'm fixing a software defect that occurs 1 in n test runs. If I want to know that the probability of it being fixed is >= p for some 0 <= p < 1, how many times, m, do I need to run the test successfully (without the defect occurring)?
3
-
$\begingroup$ This is a good question for somewhere else, but it's off-topic for MO. Try the other sites mentioned in the FAQ. $\endgroup$– Douglas ZareCommented May 7, 2010 at 19:21
-
$\begingroup$ I believe this is an ill-posed question, contrary to the implications of the previous comment and the 3 answers at the time of this writing. Regardless of the values of n, p, and m, there is no way to know, in any sense of the word, that the probability that the defect has been fixed is >= p. (Rather, in this situation statisticians make use of confidence intervals.) $\endgroup$– Daniel AsimovCommented May 9, 2010 at 19:54
-
$\begingroup$ Good point. No assumption was stated about the probability before any testing. That's fixable, or one could ask for a p-value. $\endgroup$– Douglas ZareCommented May 10, 2010 at 18:54
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
5
According to my statistics final which I took yesterday, the answer should be
$m=\lceil 2\left(1-\frac{1}{n}\right)\text{InverseErf}^2[1-p]\rceil$ where InverseErf[x] is the Inverse Error Function.
-
$\begingroup$ I'm not sure you can state a definitive answer like that without making more assumptions than the original asker states. What were your additional assumptions? $\endgroup$ Commented May 7, 2010 at 16:18
-
$\begingroup$ I was assuming that the number of times before the error occurs is roughly normally distributed - if the test is done enough times, a binomial distribution can be well-approximated by a normal distribution. $1-\text{Erf}\left[\sqrt{\frac{t\left(\frac{1}{n}-0\right)}{2 \frac{1}{n} \left(1-\frac{1}{n}\right)}}\right]$ is the equation for the probability of making a type-1 error (mistakenly assuming that we've fixed it). $\endgroup$ Commented May 7, 2010 at 16:22
-
$\begingroup$ Yes, that's a reasonable assumption. $\endgroup$ Commented May 7, 2010 at 16:30
-
$\begingroup$ The number of successes before first failure is given by a geometric distribution. It's nothing like a normal distribution. en.wikipedia.org/wiki/Geometric_distribution $\endgroup$ Commented May 7, 2010 at 18:03
-
$\begingroup$ I agree that a normal approximation is likely to be bad. If you must use a normal approximation, use $(-\infty,\frac 12)$ instead of $(-\infty, 0)$ because the count takes values in the integers. $\endgroup$ Commented May 7, 2010 at 19:12
$\begingroup$
$\endgroup$
1
I guess $m>\log(1-p)/\log(1-1/n)$ works since the probability of a faulty system running $m$ times without defect is $(1-1/n)^m$ and this should be smaller than $1-p$.
This seems to be a homework type question (and moreover an easy one) rather than a MO-question.
answered May 7, 2010 at 16:00
-
$\begingroup$ Well, it's not homework. I wish it were. It's a bug that only QA seems to be able to reproduce. And, yes, I probably should have just looked it up, since it's a rather simple question. $\endgroup$ Commented May 7, 2010 at 16:07
$\begingroup$
$\endgroup$
If your problem was a little bit more difficult (roland-bacher already provided an easy, precise and correct solution) and your $n$ is big, you can also approximate the binomial distribution by a Poisson distribution. Repeating the test $m$ times gives then the parameter $\lambda = \frac{m}{n}$ and your goal is that $m$ is big enough that $e^{-\lambda}\ge 1-p$. So $m \ge -n\ln(1-p)$. The approximation by the Poisson distribution is pretty good, for $n = 500$ and $p = 0.99$ it yields $m > 2302.585$ instead of the correct $m > 2300.28$ given by roland-bacher's formula.