Given a standard Gaussian matrix $X\in\mathbb{R}^{n\times d}$, $d<n$, with entries sampled i.i.d. from $\mathcal{N}(0,1)$, is the corresponding orthogonal projection $X X^+ = X (X^\top X)^{-1} X^\top$ a random projector, i.e., a projection in $\mathbb{R}^n$ onto a random $d$-dimensional subspace uniformly distributed in the Grassmann manifold $G_{n,d}$?

Same question if now $X\in\mathbb{R}^{n\times d}$, $d<n$, has rows sampled i.i.d. from a multivariate Gaussian distribution $\mathcal{N}(0,\Sigma)$, where $\Sigma\in\mathbb{R}^{d\times d}$ is a covariance matrix.

If not, can we say anything about concentration bounds on the quantity $\|X X^+ y\|_2$, where $y\in\mathbb{R}^n$ is a given fixed vector?


1 Answer 1


$\newcommand\R{\Bbb R}\newcommand\Si{\Sigma}$The answer to your both questions is yes, regardless of what the covariance matrix $\Sigma$ is.

Indeed, let $P_X:=X(X^\top X)^{-1}X^\top$, the orthoprojector onto $V_X:=X\R^d$, the column space of the random matrix $X$, so that $P_X\R^n=V_X$. Then for any orthogonal matrix $Q\in\R^{n\times n}$ we have $P_{QX}=QX(X^\top X)^{-1}X^\top Q^\top$ and hence $$V_{QX}=QX(X^\top X)^{-1}X^\top Q^\top\R^n =QX(X^\top X)^{-1}X^\top\R^n=QV_X.$$ Therefore and because $QX$ equals $X$ in distribution (see the details on this below), it follows that $V_X$ equals $QV_X$ in distribution, for any orthogonal matrix $Q\in\R^{n\times n}$. So, $V_X$ is uniformly distributed (over $G_{n,d}$ if $\Si$ is nonsingular, as it is usually assumed, or, more generally, over $G_{n,r}$ where $r$ is the rank of $\Si$). $\quad\Box$

Details on why $QX$ equals $X$ in distribution, for any orthogonal matrix $Q\in\R^{n\times n}$: Write $X=[X_{ij}]_{i\in[n],j\in[d]}$ and $Q=[Q_{ij}]_{i\in[n],j\in[n]}$, where $[n]:=\{1,\dots,n\}$. Then the entries $(QX)_{ij}=\sum_{k\in[n]}Q_{ik}X_{k,j}$ of the matrix $QX$ are zero-mean jointly normal random variables, and for all $i,i'$ in $[n]$ and $j,j'$ in $[d]$ we have $$E(QX)_{ij}(QX)_{i'j'} =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}EX_{kj}X_{k'j'} \\ =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}1(k=k')\Si_{jj'} =\sum_{k\in[n]}Q_{ik}Q_{i'k}\Si_{jj'} =1(i=i')\Si_{jj'},$$ so that the covariances of the $(QX)_{ij}$'s do not depend on $Q$.

  • $\begingroup$ What if $\det\Sigma=0 \text{?} \qquad$ $\endgroup$ Commented Jun 6 at 20:42
  • 1
    $\begingroup$ @MichaelHardy : I have added a detail on this. $\endgroup$ Commented Jun 6 at 21:29
  • $\begingroup$ Noted. One should also note that the case $\det \Sigma = 0$ arises very naturally in some contexts, perhaps the simplest of which is that if $X_1,\ldots,X_n\sim\text{i.i.d.} \operatorname N_1(\mu,\sigma^2)$ so that $\overline X = (X_1 + \cdots + X_n)/n \sim \operatorname N(\mu, \sigma^2/n),$ then the vector whose $i$th component is $X_i - \overline X$ for $i=1,\ldots,n$ has a singular covariance matrix, and one uses that to show that $\sum_{i=1}^n (X_i-\overline X)^2 \sim \sigma^2 \chi_{n-1}^2,$ and if$\,\ldots\qquad$ $\endgroup$ Commented Jun 6 at 21:47
  • $\begingroup$ $\ldots X_i \sim \operatorname N(a + bw_i, \sigma^2)$ and $\widehat a,\widehat {b\,}$ are the least-squares estimators of $a,b$ then $\sum_{i=1}^n \left(X_i - (\widehat a + \widehat {b\,} w_i)\right)^2 \sim \sigma^2 \chi_{n-2}^2$ and that that sum is probabilistically independent of the least-squares estimators $\widehat a,\widehat{b\,}.$ $\endgroup$ Commented Jun 6 at 21:49
  • $\begingroup$ @MichaelHardy : I agree with your latter comments. $\endgroup$ Commented Jun 6 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.