On Impossible events

Question

Let's consider a continuous random variable $X$ distributed according to a PDF $p(x):\mathbb{R}\mapsto \mathbb{R}_{\geq 0}$.

Is there a meaningful sense in which one could say that for any $x_0:p(x_0)=0$ the event $X=x_0$ is impossible? By impossible, I mean that if we simulate $X$ then we never observe in the sample the value $x_0$.

If the answer is no, how do we make sure that $X$ will never get the value $x_0$?

Note that my question is not about the well-known fact that any event of the form $X=x_0$ is almost impossible, in the sense that $\mathbb{P}(X=x_0)=0$, and this does not mean that such events cannot happen (i.e., if we simulate $X$ then we can observe the value $x_0$ in the sample).

Answer 1

This is a good question about practical simulation, but it needs a bit of reinterpretation. This is what I understand to be the question(s):

Q1. What is meant by saying that we are sampling finitely represented numbers (e.g. IEEE floating point) from a continuous distribution, given by a density function which is zero at some points? For example, sampling from the uniform distribution over the closed interval $[0,1]$ or the open interval $(0,1)$.

Q2: How is the intended sampling achieved in practice?

For starters, recall that you cannot exactly simulate from a continuous distribution anyway, in a computer with bounded storage and bounded time. This holds even if the density is nowhere zero, say, the standard normal distribution: your simulated values will be in some finite set, whether that is IEEE floating point, or something else.

So there are in fact two (related) distributions here: The underlying continuous distribution, and the discrete sampling distribution of the values that you are really receiving.

Answer to Q1

The canonical example is simulating from the uniform distribution over the open interval $(0,1)$, or the half-open interval $[0,1)$ or $(0,1]$. Simulating from $\mathrm{Unif}(0,1]$, what you typically get is an IEEE floating point number. If you take it at face value, it is a rational number, from a particular finite subset of $\mathbb{Q}$. Among other things this set includes integers of some bounded size. Your next arithmetic operation might well be (say) $1/x$, and it would be inconvenient if $x=0$ has positive probability.

Note that mathematically there is no difference in "the uniform over $(0,1)$" and "the uniform over $[0,1]$". They specify the same cumulative distribution function. So there is no difference in sampling from this or that (because they are the same thing).

What is meant is a difference in the sampling distribution, not in the underlying continuous distribution. I believe it is usually meant, and desired, that the sampling should never return a (floating) number $x_0$ where the underlying density is zero.

In particular, by saying "sample from the uniform over $(0,1)$", we mean "sample from that distribution so that the samples are in the interval $(0,1)$". Similar for closed and half-open intervals. This is a convenient mathematical fiction, or abuse of notation.

Answer to Q2

Actual pseudorandom generators do take the endpoints into account. For example, the Python random says it will return a "random floating point number in the range 0.0 <= X < 1.0", and that's what it does. The implementation is not that difficult: First it generates a pseudorandom integer in range $0,\ldots,2^{53}-1 = 9007199254740991$. Then divides by $2^{53}$. The precision has been chosen so that the maximum possible value is a float strictly smaller than $1.0$. So it never returns $1.0$, not even with a small probability.

If you want to simulate from $\mathrm{Unif}(0,1)$ with both endpoints impossible, there are two simple solutions. One is to reject and resample if the undesired value happens (which is rare). The other is to sample the integer from $1,\dots,2^{53}-1$ in the first place.

Endpoints are usually the main concern here, and they are easily dealt with. Another typical endpoint case is sampling a positive waiting time from $\mathrm{Exp}(\lambda)$.

Much less common would be isolated zeros in the density, such as $f(x) = (3/2)x^2$ for $-1 < x < 1$. If you don't want $x=0$ to ever happen in your simulation, the obvious easy solution is rejection and resampling in that rare case. There are other solutions that avoid resampling: you could first sample from the strictly positive part, then flip the sign with probability $0.5$.

Note: A related question about the implementation has been asked and answered in Cross Validated.