distance from the mean of a normal distribution to the span of a random sample

Let $W$ be a $d\times k$ matrix whose columns are sampled from a multivariate normal distribution with mean $\mu$ and unit covariance. I'm interested in $|\mu - WW^+\mu|$, that is the distance from the mean to the subspace spanned by the samples.

This seems like the sort of problem that should already be solved somewhere, but I don't know where to look. Everything I've found on random projections assumes a centered distribution, but what makes this problem interesting is the non-zero mean.

• In the light of Josu E. M.'s answer below the question may benefit from some more specification on if you expect $k$ to be smaller or larger than $d$. To me it sounds $d$ is fixed and we can go on and on sampling making $k$ arbitrary large, but as per the answer below only the $k < d$ case is interesting. Is there a specific application you had in mind? Aug 10 '16 at 11:36
• Yes, I'm assuming that $k<d$. Aug 30 '16 at 17:47

As the $W$ matrix entries follow a multivariate normal distribution, then the probability that this matrix is singular is zero.

That makes the matrix full rank, implying that $W^\dagger = W^{-1}$. That makes the last distance be zero: $|\mu- WW^\dagger\mu| = |\mu - I\mu| = 0$.

As Vincent stated in his comment, the interesting case is when $k<d$, and I realized that for that case the result changes as follows:

$|\mu-WW^\dagger\mu|=|(I-U\Sigma V^TV\Sigma^\dagger U^T)\mu|=|U(I-\Sigma\Sigma^\dagger)U^T\mu|$

Then the product $\Sigma\Sigma^\dagger$ will give us a $dxd$ matrix of the next form:

$\Sigma\Sigma^\dagger = \left( \begin{array}{ccc} I_{kxk} & 0\\ 0 & 0\\ \end{array} \right)$

due to the fact that the matrix is full rank of rank k. That makes the next thing happen: $\xi=(I - \Sigma\Sigma^\dagger) = \left( \begin{array}{ccc} 0 & 0\\ 0 & I_{(d-k)x(d-k)}\\ \end{array} \right)$

so and as vector norm is unitarily invariant:

$|U\xi U^T\mu|=|\xi U^T \mu|$

and then=

$|\xi U^T \mu| = \left| \left( \begin{array}{ccc} 0 \\ u_{j>k}^T\\ \end{array} \right)\mu\right| =\left| \left( \begin{array}{ccc} 0 \\ u_{j>k}^T\mu\\ \end{array} \right)\right| = \sum_{j>k}(u_j^T\mu)^2$

So we get that value for the distance asked.

• This doesn't actually answer the question for me. You haven't used anywhere the random structure of the $W$ matrix, and the final answer is still a function of the particular values of $W$. (Sorry for the delayed response as I've been on vacation.) Aug 30 '16 at 17:46