Skip to main content
2 of 2
replaced http://math.stackexchange.com/ with https://math.stackexchange.com/

It means pretty much the same thing as when we say that we have measured a physical property to a certain accuracy. That is, in loose terms, it means that we are reasonably certain that the correct value lies close to the estimated value — and, if we're to be proper, we should also state how close, how certain we are of this, and how this certainty was estimated.

As Bjørn Kjos-Hanssen already noted, this (un)certainty is typically expressed by a confidence interval, which basically says that we are, for example, 95% (or 99.9%, or whatever) sure that the true value lies within this interval.

As a specific example, let us say that we have sampled 100 billion (hopefully) independent and uniformly distributed Monte Carlo realizations of some process, and observed that a certain property is satisfied in 31,895,060,547 of those samples. Each of these samples can be regarded as a Bernoulli trial succeeding with some unknown probability $p$, which we wish to estimate. Thus, the number of successes in $n$ samples is binomially distributed with parameters $n$ and $p$; $n$ is known, and we wish to estimate and characterize the distribution of $p$, given the number of observed successes.

We could now proceed in several ways:

  • We could directly apply Bayes' theorem to calculate the conditional distribution of $p$, given the observed number of successes (which will typically be a Beta distribution, if we were smart enough to pick one as our assumed prior distribution of $p$), and then summarize it e.g. by calculating a credible interval for $p$ at whatever confidence level we consider reasonable.

  • We could skip the Bayesian calculation, and just directly apply one of the standard formulae for a binomial proportion confidence interval, such as the Wilson or the Jeffreys interval.

  • Since we have a lot of samples, and a reasonable proportion of successes, we could simply approximate the distribution of $p$ by a normal distribution having mean equal to the sample mean $\hat p = \frac kn$, where $k$ is the number of observed successes, and variance $\sigma^2 = \frac1n\hat p(1-\hat p)$, and then just quote $\hat p \pm 2\sigma$ as our (slightly conservative) 95% confidence interval for $p$.

Using any of these methods, for the example above, we can obtain a 95% confidence interval of $p \in 0.318951 \pm 2.9\times10^{-6}$ or so. If we wanted to be really sure of getting the true value within the interval, we could widen it to, say $\pm 6.11\sigma = 9.0\times10^{-6}$, and be 99.9999999% sure that the true value of $p$ lies within the interval. (Of course, at those confidence levels, we should start worrying about the non-zero probability of undetected bugs in our Monte Carlo code, or in the random number generator it's using, or in the computer hardware it we ran it on, or in the theory it's based on...)

In fact, for quick and dirty estimation of the accuracy of binomial Monte Carlo results, we can simplify the normal approximation further by noting that $(1-\hat p) \le 1$, and so $n\sigma$ $=$ $\sqrt{n\hat p(1-\hat p)}$ $\le$ $\sqrt{n\hat p}$ $=$ $\sqrt k$. Thus, if we observe a $d$-digit number of successes among our trials, we may roughly estimate that the resulting estimate of the success probability has about $d/2$ significant digits, with the remaining digits being essentially random.