3
$\begingroup$

Given $s$ successes in $n$ trials, where $p=\frac{s}{n}$, is there a standard way to determine if I have enough data to compute a meaningful statistic? For example, given $s=1, n=10, p=0.1$, the 95% confidence interval ranges from $0.002 < p < 0.445$.

It seems like I could just use the gap between the 95% confidence interval, but it falls apart with rare events (ie: $\frac{s=1}{n=10}$ vs $\frac{s=10}{n=100}$), the 95% confidence of the one on the left is +/- 0.345 while the one on the right is +/- 0.045, yet relative to the estimated probability they are the about the same.

Since I am using the estimated probability to tell if a process is in control in the context of it's historical trend, I don't want to raise an alert on an outlier that's caused by too little data.

Am I trying to solve this the wrong way?

I am very grateful for any guidance that points me in the right direction. Thanks for reading!

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
2
$\begingroup$

It doesn't seem clear how you got your alleged 95% confidence interval. Whether you're going about it the wrong way would be easier to assess if you told us how you're going about it.

The most frequently taught method of finding a confidence interval for $p$ works reasonably well when the observed number of successes and failures are reasonably large, but in your case you've observed only one success. That method says that the expected value and variance of the number of successes in $n$ independent trials with probability $p$ of success on each trial are respectively $np$ and $npq=np(1-p)$, and then approximates the distribution by a normal distribution with that mean and that variance, using $s/n$ as the value of $p$ estimated from the sample. The mean and variance of the distribution of the proportion of successes are $p$ and $pq/n$. Thus the endpoints of a 95% confidence interval would be $$ \frac{s}{n} \pm 1.96 \sqrt{\frac{(s/n)(1-s/n)}{n}}. $$ In your case the interval given by this formula is $$ -0.086 < p < 0.286 $$ and of course seeing a negative number there should tell you that this isn't very reliable, which you expect since you've observed only one success. But this leaves the question of how you got $0.002$ and $0.445$? Also, have you got something like a null hypothesis based on that historical trend? Conceivably you'd also want to use other prior information.

(Also, stats.stackexchange.com would probably give you more information than this forum would.)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi Michael, Thank you for your thoughtful response. I'm using R function binom.test to compute the 95% confidence interval as I'm prototyping now, but am going to need to re-code my solution and will end up using the same formula you listed up above, thank you :) As far as the null hypothesis goes, I'm calculating the daily proportion of daily successes in a defined body of samples. I suppose the null hypothesis would be, in this instance, that there is not enough data to trust the calculation. If my average proportion was 0.1%, and on one day I had 0 leads of 500, I.... $\endgroup$
    – Tim Harper
    Commented Jun 22, 2011 at 14:41
  • $\begingroup$ wouldn't want to use that data to impact my control chart or trigger an alert. (I'm writing a data monitoring system) $\endgroup$
    – Tim Harper
    Commented Jun 22, 2011 at 14:42
  • $\begingroup$ The manual entry for binom.test says it does an exact test, so that's not what I was talking about above. Just exactly what it does is not yet clear to me. $\endgroup$
    – Michael Hardy
    Commented Jun 22, 2011 at 23:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .