Let $W$ be a $d\times k$ matrix whose columns are sampled from a multivariate normal distribution with mean $\mu$ and unit covariance. I'm interested in $|\mu - WW^+\mu|$, that is the distance from the mean to the subspace spanned by the samples.
This seems like the sort of problem that should already be solved somewhere, but I don't know where to look. Everything I've found on random projections assumes a centered distribution, but what makes this problem interesting is the non-zero mean.
As the $W$ matrix entries follow a multivariate normal distribution, then the probability that this matrix is singular is zero.
That makes the matrix full rank, implying that $W^\dagger = W^{-1}$. That makes the last distance be zero: $|\mu- WW^\dagger\mu| = |\mu - I\mu| = 0$.
As Vincent stated in his comment, the interesting case is when $k<d$, and I realized that for that case the result changes as follows:
$|\mu-WW^\dagger\mu|=|(I-U\Sigma V^TV\Sigma^\dagger U^T)\mu|=|U(I-\Sigma\Sigma^\dagger)U^T\mu|$
Then the product $\Sigma\Sigma^\dagger$ will give us a $dxd$ matrix of the next form:
$\Sigma\Sigma^\dagger = \left( \begin{array}{ccc} I_{kxk} & 0\\ 0 & 0\\ \end{array} \right) $
due to the fact that the matrix is full rank of rank k. That makes the next thing happen: $\xi=(I - \Sigma\Sigma^\dagger) = \left( \begin{array}{ccc} 0 & 0\\ 0 & I_{(d-k)x(d-k)}\\ \end{array} \right) $
so and as vector norm is unitarily invariant:
$|U\xi U^T\mu|=|\xi U^T \mu|$
and then=
$|\xi U^T \mu| = \left| \left( \begin{array}{ccc} 0 \\ u_{j>k}^T\\ \end{array} \right)\mu\right| =\left| \left( \begin{array}{ccc} 0 \\ u_{j>k}^T\mu\\ \end{array} \right)\right| = \sum_{j>k}(u_j^T\mu)^2 $
So we get that value for the distance asked.
