# Orthogonal projection $X X^+$ from random Gaussian matrix $X$

Given a standard Gaussian matrix $$X\in\mathbb{R}^{n\times d}$$, $$d, with entries sampled i.i.d. from $$\mathcal{N}(0,1)$$, is the corresponding orthogonal projection $$X X^+ = X (X^\top X)^{-1} X^\top$$ a random projector, i.e., a projection in $$\mathbb{R}^n$$ onto a random $$d$$-dimensional subspace uniformly distributed in the Grassmann manifold $$G_{n,d}$$?

Same question if now $$X\in\mathbb{R}^{n\times d}$$, $$d, has rows sampled i.i.d. from a multivariate Gaussian distribution $$\mathcal{N}(0,\Sigma)$$, where $$\Sigma\in\mathbb{R}^{d\times d}$$ is a covariance matrix.

If not, can we say anything about concentration bounds on the quantity $$\|X X^+ y\|_2$$, where $$y\in\mathbb{R}^n$$ is a given fixed vector?

$$\newcommand\R{\Bbb R}\newcommand\Si{\Sigma}$$The answer to your both questions is yes, regardless of what the covariance matrix $$\Sigma$$ is.
Indeed, let $$P_X:=X(X^\top X)^{-1}X^\top$$, the orthoprojector onto $$V_X:=X\R^d$$, the column space of the random matrix $$X$$, so that $$P_X\R^n=V_X$$. Then for any orthogonal matrix $$Q\in\R^{n\times n}$$ we have $$P_{QX}=QX(X^\top X)^{-1}X^\top Q^\top$$ and hence $$V_{QX}=QX(X^\top X)^{-1}X^\top Q^\top\R^n =QX(X^\top X)^{-1}X^\top\R^n=QV_X.$$ Therefore and because $$QX$$ equals $$X$$ in distribution (see the details on this below), it follows that $$V_X$$ equals $$QV_X$$ in distribution, for any orthogonal matrix $$Q\in\R^{n\times n}$$. So, $$V_X$$ is uniformly distributed (over $$G_{n,d}$$ if $$\Si$$ is nonsingular, as it is usually assumed, or, more generally, over $$G_{n,r}$$ where $$r$$ is the rank of $$\Si$$). $$\quad\Box$$
Details on why $$QX$$ equals $$X$$ in distribution, for any orthogonal matrix $$Q\in\R^{n\times n}$$: Write $$X=[X_{ij}]_{i\in[n],j\in[d]}$$ and $$Q=[Q_{ij}]_{i\in[n],j\in[n]}$$, where $$[n]:=\{1,\dots,n\}$$. Then the entries $$(QX)_{ij}=\sum_{k\in[n]}Q_{ik}X_{k,j}$$ of the matrix $$QX$$ are zero-mean jointly normal random variables, and for all $$i,i'$$ in $$[n]$$ and $$j,j'$$ in $$[d]$$ we have $$E(QX)_{ij}(QX)_{i'j'} =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}EX_{kj}X_{k'j'} \\ =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}1(k=k')\Si_{jj'} =\sum_{k\in[n]}Q_{ik}Q_{i'k}\Si_{jj'} =1(i=i')\Si_{jj'},$$ so that the covariances of the $$(QX)_{ij}$$'s do not depend on $$Q$$.
• What if $\det\Sigma=0 \text{?} \qquad$ Commented Jun 6 at 20:42
• Noted. One should also note that the case $\det \Sigma = 0$ arises very naturally in some contexts, perhaps the simplest of which is that if $X_1,\ldots,X_n\sim\text{i.i.d.} \operatorname N_1(\mu,\sigma^2)$ so that $\overline X = (X_1 + \cdots + X_n)/n \sim \operatorname N(\mu, \sigma^2/n),$ then the vector whose $i$th component is $X_i - \overline X$ for $i=1,\ldots,n$ has a singular covariance matrix, and one uses that to show that $\sum_{i=1}^n (X_i-\overline X)^2 \sim \sigma^2 \chi_{n-1}^2,$ and if$\,\ldots\qquad$ Commented Jun 6 at 21:47
• $\ldots X_i \sim \operatorname N(a + bw_i, \sigma^2)$ and $\widehat a,\widehat {b\,}$ are the least-squares estimators of $a,b$ then $\sum_{i=1}^n \left(X_i - (\widehat a + \widehat {b\,} w_i)\right)^2 \sim \sigma^2 \chi_{n-2}^2$ and that that sum is probabilistically independent of the least-squares estimators $\widehat a,\widehat{b\,}.$ Commented Jun 6 at 21:49