3
$\begingroup$

Given a standard Gaussian matrix $X\in\mathbb{R}^{n\times d}$, $d<n$, with entries sampled i.i.d. from $\mathcal{N}(0,1)$, is the corresponding orthogonal projection $X X^+ = X (X^\top X)^{-1} X^\top$ a random projector, i.e., a projection in $\mathbb{R}^n$ onto a random $d$-dimensional subspace uniformly distributed in the Grassmann manifold $G_{n,d}$?

Same question if now $X\in\mathbb{R}^{n\times d}$, $d<n$, has rows sampled i.i.d. from a multivariate Gaussian distribution $\mathcal{N}(0,\Sigma)$, where $\Sigma\in\mathbb{R}^{d\times d}$ is a covariance matrix.

If not, can we say anything about concentration bounds on the quantity $\|X X^+ y\|_2$, where $y\in\mathbb{R}^n$ is a given fixed vector?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

$\newcommand\R{\Bbb R}\newcommand\Si{\Sigma}$The answer to your both questions is yes, regardless of what the covariance matrix $\Sigma$ is.

Indeed, let $P_X:=X(X^\top X)^{-1}X^\top$, the orthoprojector onto $V_X:=X\R^d$, the column space of the random matrix $X$, so that $P_X\R^n=V_X$. Then for any orthogonal matrix $Q\in\R^{n\times n}$ we have $P_{QX}=QX(X^\top X)^{-1}X^\top Q^\top$ and hence $$V_{QX}=QX(X^\top X)^{-1}X^\top Q^\top\R^n =QX(X^\top X)^{-1}X^\top\R^n=QV_X.$$ Therefore and because $QX$ equals $X$ in distribution (see the details on this below), it follows that $V_X$ equals $QV_X$ in distribution, for any orthogonal matrix $Q\in\R^{n\times n}$. So, $V_X$ is uniformly distributed (over $G_{n,d}$ if $\Si$ is nonsingular, as it is usually assumed, or, more generally, over $G_{n,r}$ where $r$ is the rank of $\Si$). $\quad\Box$

Details on why $QX$ equals $X$ in distribution, for any orthogonal matrix $Q\in\R^{n\times n}$: Write $X=[X_{ij}]_{i\in[n],j\in[d]}$ and $Q=[Q_{ij}]_{i\in[n],j\in[n]}$, where $[n]:=\{1,\dots,n\}$. Then the entries $(QX)_{ij}=\sum_{k\in[n]}Q_{ik}X_{k,j}$ of the matrix $QX$ are zero-mean jointly normal random variables, and for all $i,i'$ in $[n]$ and $j,j'$ in $[d]$ we have $$E(QX)_{ij}(QX)_{i'j'} =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}EX_{kj}X_{k'j'} \\ =\sum_{k,k'\in[n]}Q_{ik}Q_{i'k'}1(k=k')\Si_{jj'} =\sum_{k\in[n]}Q_{ik}Q_{i'k}\Si_{jj'} =1(i=i')\Si_{jj'},$$ so that the covariances of the $(QX)_{ij}$'s do not depend on $Q$.

Improve this answer
$\endgroup$
7
  • $\begingroup$ What if $\det\Sigma=0 \text{?} \qquad$ $\endgroup$
    – Michael Hardy
    Commented Jun 6 at 20:42
  • 1
    $\begingroup$ @MichaelHardy : I have added a detail on this. $\endgroup$
    – Iosif Pinelis
    Commented Jun 6 at 21:29
  • $\begingroup$ Noted. One should also note that the case $\det \Sigma = 0$ arises very naturally in some contexts, perhaps the simplest of which is that if $X_1,\ldots,X_n\sim\text{i.i.d.} \operatorname N_1(\mu,\sigma^2)$ so that $\overline X = (X_1 + \cdots + X_n)/n \sim \operatorname N(\mu, \sigma^2/n),$ then the vector whose $i$th component is $X_i - \overline X$ for $i=1,\ldots,n$ has a singular covariance matrix, and one uses that to show that $\sum_{i=1}^n (X_i-\overline X)^2 \sim \sigma^2 \chi_{n-1}^2,$ and if$\,\ldots\qquad$ $\endgroup$
    – Michael Hardy
    Commented Jun 6 at 21:47
  • $\begingroup$ $\ldots X_i \sim \operatorname N(a + bw_i, \sigma^2)$ and $\widehat a,\widehat {b\,}$ are the least-squares estimators of $a,b$ then $\sum_{i=1}^n \left(X_i - (\widehat a + \widehat {b\,} w_i)\right)^2 \sim \sigma^2 \chi_{n-2}^2$ and that that sum is probabilistically independent of the least-squares estimators $\widehat a,\widehat{b\,}.$ $\endgroup$
    – Michael Hardy
    Commented Jun 6 at 21:49
  • $\begingroup$ @MichaelHardy : I agree with your latter comments. $\endgroup$
    – Iosif Pinelis
    Commented Jun 6 at 21:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .