Sample from uniform distribution vs. Sample from random distribution - MathOverflow

Sample from uniform distribution vs. Sample from random distribution

2010-01-08T14:54:02Z

I could sample a set of m elements from the uniform distribution over a universe $U$ of n >> m elements. Alternately, I could select a random probability distribution $\mathcal{D}$, and sample $m$ elements from $\mathcal{D}$.

EDIT (per Michael Lugo): When I say "select a random probability distribution", I mean select a point uniformly at random from the standard $n$-simplex: {${(x_1,\ldots,x_n) : x_i \geq 0, x_1+\ldots+x_n = 1}$}.

Do these two methods lead to the same distribution over my sample? If not, how do they differ? If some event (my sample lies in some set of samples of size m) occurs with probability p using the second method, what can I say about its probability using the first method?

Answer by Mark Meckes for Sample from uniform distribution vs. Sample from random distribution

2010-01-08T16:17:43Z

You're right that things change for m>1; I was thinking sloppily.

Assume $U=\{1,\ldots,n\}$ for concreteness. If $Y_1,\ldots,Y_m$ are chosen independently and uniformly from $U$, then for any $k_1,\ldots,k_m\in U$, we of course have $$ \Pr[Y_1=k_1,\ldots,Y_m=k_m] = \frac{1}{n^m}. $$

On the other hand, if $x=(x_1,\ldots,x_m)$ is chosen uniformly from the standard $n$-simplex and $Y_1,\ldots,Y_m$ are then chosen independently according to $x$, then $$ \Pr[Y_1=k_1,\ldots,Y_m=k_m] = \mathbb{E}\Pr[Y_1=k_1,\ldots,Y_m=k_m|x] = \mathbb{E}\prod_{i=1}^m x_{k_i} = \frac{n!}{(n+r)!}\prod_{j=1}^n r_j!, $$ where $r_j = \#\{1\le i \le m : k_i=j\}$ and $r=r_1 + \cdots r_n$. This last expectation can be proved most easily from Lemma 1 in this paper.

Answer by Douglas Zare for Sample from uniform distribution vs. Sample from random distribution

2010-01-24T17:43:02Z

While the exact answer by Mark Meckes is nice, it's worth pointing out that if you condition on not repeating elements, the conditional distributions are equal by symmetry, and your condition $n \gt\gt m$ is close to what you need to say that repetitions are rare.

Repetition is much more common if you choose a random weighting and then sample from that instead of sample uniformly. The condition that $n$ is much greater than $m^2$ means that repetition is rare in samples from the uniform distribution, as the expected number of repetitions of pairs $Y_i = Y_j$ is $\binom{m}{2}/n$.

If we choose a random distribution, the weight on a particular element follows a beta distribution $\beta(1,n-1)$. The probability that both $Y_i$ and $Y_y$ equal that element is the 2nd moment, variance + mean^2, or $(n-1)/(n^2 (n+1)) + 1/n^2 = 2/(n(n+1))$. The probability $P_2(Y_i=Y_j)=2/(n+1)$ instead of $1/n$, and the expected number of repetitions of pairs is $2\binom{m}{2}/(n+1)$.

If $n \gt\gt m^2$, there is low total variation distance. Let $\Delta$ be the diagonal set where there is at least one repetition. $|P_2(S)-P_1(S)| \le 2P_2(\Delta) \le 4\binom{m}{2}/(n+1) < 2m^2/n$.