Let $W$ be a $d\times k$ matrix whose columns are sampled from a multivariate normal distribution with mean $\mu$ and unit covariance. I'm interested in $|\mu - WW^+\mu|$, that is the distance from the mean to the subspace spanned by the samples.

This seems like the sort of problem that should already be solved somewhere, but I don't know where to look. Everything I've found on random projections assumes a centered distribution, but what makes this problem interesting is the non-zero mean.

  • $\begingroup$ In the light of Josu E. M.'s answer below the question may benefit from some more specification on if you expect $k$ to be smaller or larger than $d$. To me it sounds $d$ is fixed and we can go on and on sampling making $k$ arbitrary large, but as per the answer below only the $k < d$ case is interesting. Is there a specific application you had in mind? $\endgroup$
    – Vincent
    Aug 10 '16 at 11:36
  • $\begingroup$ Yes, I'm assuming that $k<d$. $\endgroup$ Aug 30 '16 at 17:47

As the $W$ matrix entries follow a multivariate normal distribution, then the probability that this matrix is singular is zero.

That makes the matrix full rank, implying that $W^\dagger = W^{-1}$. That makes the last distance be zero: $|\mu- WW^\dagger\mu| = |\mu - I\mu| = 0$.

As Vincent stated in his comment, the interesting case is when $k<d$, and I realized that for that case the result changes as follows:

$|\mu-WW^\dagger\mu|=|(I-U\Sigma V^TV\Sigma^\dagger U^T)\mu|=|U(I-\Sigma\Sigma^\dagger)U^T\mu|$

Then the product $\Sigma\Sigma^\dagger$ will give us a $dxd$ matrix of the next form:

$\Sigma\Sigma^\dagger = \left( \begin{array}{ccc} I_{kxk} & 0\\ 0 & 0\\ \end{array} \right) $

due to the fact that the matrix is full rank of rank k. That makes the next thing happen: $\xi=(I - \Sigma\Sigma^\dagger) = \left( \begin{array}{ccc} 0 & 0\\ 0 & I_{(d-k)x(d-k)}\\ \end{array} \right) $

so and as vector norm is unitarily invariant:

$|U\xi U^T\mu|=|\xi U^T \mu|$

and then=

$|\xi U^T \mu| = \left| \left( \begin{array}{ccc} 0 \\ u_{j>k}^T\\ \end{array} \right)\mu\right| =\left| \left( \begin{array}{ccc} 0 \\ u_{j>k}^T\mu\\ \end{array} \right)\right| = \sum_{j>k}(u_j^T\mu)^2 $

So we get that value for the distance asked.

  • $\begingroup$ This doesn't actually answer the question for me. You haven't used anywhere the random structure of the $W$ matrix, and the final answer is still a function of the particular values of $W$. (Sorry for the delayed response as I've been on vacation.) $\endgroup$ Aug 30 '16 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.