Asymptotic property of the left singular vectors of i.i.d. data matrix

Question

Let $\mathbf{X}$ be $(n \times p)$-dimensional data matrix ($n > p$) whose rows $\mathbf{x}_i$ are i.i.d. with some finite moments: $$ \mathbf{X}^\top = [\mathbf{x}_1, \ldots \mathbf{x}_n]^\top. $$ By the singular decomposition, we can obtain $$ \mathbf{X} = \mathbf{U} \mathbf{D} \mathbf{V}^\top = \mathbf{U}_{:r} \mathbf{D}_{:r} \mathbf{V}_{:r}^\top, $$ where $r$ is the rank of $\mathbf{X}$ which may $r<p$. Notice that here I used a Matlab colon notation ":" to denote submatrices \begin{align*} \mathbf{U}_{:r} &:= [\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_r], \\ \mathbf{D}_{:r} &:= \operatorname{diag}([d_1,d_2,\ldots,d_r]), \\ \mathbf{V}_{:r} &:= [\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_r], \end{align*} (Notice that $\mathbf{U}_{:r}$ is not a square matrix, so $\mathbf{U}_{:r}^\top \mathbf{U}_{:r} = \mathbf{I}$, but $\mathbf{U}_{:r}\mathbf{U}_{:r}^\top \neq \mathbf{I}$). Then the question is

Question.

As $n \to \infty$ while $p$ is fixed, what can we know about the asymptotic property of $\mathbf{U}_{:r}$, for example, the limit of $n^{-1} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y}$, where $\mathbf{y}$ is a random vector whose elements are i.i.d. and independent to $\mathbf{X}$?

As an example, we can know about the asymptotic properties of $\mathbf{D}_{:r}$ and $\mathbf{V}_{:r}$: by the law of large numbers, as $n \to \infty$ (while $p$ is fixed), $$ \frac{1}{n} \mathbf{X}^\top \mathbf{X} = \frac{1}{n} \mathbf{V}_{:r} \mathbf{D}_{:r}^2 \mathbf{V}_{:r}^\top \xrightarrow{\mathbb{P}} \mathbb{E} \big[ \mathbf{x}_i \mathbf{x}_i^\top \big], $$ so $\mathbf{V}_{:r}$ and $\mathbf{D}_{:r}^2$ converges to the eigenvectors and eigenvalues of $\mathbb{E} \big[ \mathbf{x}_i \mathbf{x}_i^\top \big]$. Of course that since the number of rows of $\mathbf{U}$ goes to infinity, we cannot directly say what is the limit of $\mathbf{U}_{:r}$, but we may know (or at least the existence) the limit of $n^{-1} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y}$.

Here we assume that $\mathbf{X}$ and $\mathbf{y}$ have finite moments (and sub-Gaussianity).

If this question is too elementary, I really apologize for that, but this question is very important to prove the asymptotic properties of my statistical estimator. It would be really appreciated if you give any help.

Thanks,

Answer 1

This is for the self-reference.

Claim. There exists some $C>0$ such that $$\frac{1}{n} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y} \xrightarrow{\mathbb{P}} C.$$

Proof. The main idea of the proof is that linear approximation of the left singular vectors $\mathbf{U}_{:r}$ using the limit of $\mathbf{V}_{:r}$ and $\mathbf{D}_{:r}$.

(Step 1: Limit of $\mathbf{V}_{:r}$ and $\mathbf{D}_{:r}$). Since we know that $n^{-1} \mathbf{X}^\top \mathbf{X} \xrightarrow{\mathbb{P}} \mathbb{E}[\mathbf{x}_i \mathbf{x}_i^\top]$,by the Davis-Kahan theorem and Weyl's inequality, we have \begin{align*} &\mathbf{V}_{:r} = \mathbf{V}_0 \mathbf{O}^\top + O_{\mathbb{P}}\bigg(\frac{1}{\sqrt{n}}\bigg), &\frac{\mathbf{D}_{:r}^2}{n} = \frac{\mathbf{D}_{0}^2}{n} + O_{\mathbb{P}}\bigg(\frac{1}{\sqrt{n}}\bigg), \end{align*} where $\mathbf{V}_0$ and $\mathbf{D}_0^2/n$ are the first $r$ eigenvectors and eigenvalues of $\mathbb{E}[\mathbf{x}_i\mathbf{x}_i^\top]$, respectively, and $\mathbf{O}$ is some orthogonal matrix which depends on $\mathbf{V}_{:r}$ and $\mathbf{V}_0$.

(Step 2: Approximation of $\mathbf{U}_{:r}$). By using the previous result, we can approximate the left singular vectors $\mathbf{U}_{:r}$ as follows: \begin{align*} \mathbf{U}_{:r} = \mathbf{X} \mathbf{V}_{:r} \mathbf{D}_{:r}^{-1} = \mathbf{X} \mathbf{V}_0 \mathbf{O}^\top \mathbf{D}_0^{-1} + \mathbf{X} \mathbf{R} \approx \mathbf{X} \mathbf{V}_0 \mathbf{O}^\top \mathbf{D}_0^{-1}, \end{align*} where \begin{align*} \mathbf{R} &:= \mathbf{V}_{:r} \mathbf{D}_{:r}^{-1} - \mathbf{V}_0 \mathbf{O}^\top \mathbf{D}_0^{-1} \\ &= (\mathbf{V}_{:r} - \mathbf{V}_0 \mathbf{O}^\top) \mathbf{D}_{:r}^{-1} + \mathbf{V}_0 \mathbf{O}^\top (\mathbf{D}_{:r}^{-1} - \mathbf{D}_0^{-1}) \\ &= O_{\mathbb{P}}\bigg(\frac{1}{n}\bigg). \end{align*} Therefore, we have \begin{align*} \frac{1}{n} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y} &= \frac{1}{n} \mathbf{y}^\top \mathbf{X} \mathbf{V}_0 \mathbf{O} \mathbf{D}_0^{-2} \mathbf{O}^\top \mathbf{V}_0^\top \mathbf{X}^\top \mathbf{y} + O_{\mathbb{P}} \bigg(\frac{1}{\sqrt{n}}\bigg) + O_{\mathbb{P}} \bigg(\frac{1}{n}\bigg) \\ &= \frac{1}{n} \mathbf{y}^\top \mathbf{X} \mathbf{V}_0 \mathbf{D}_0^{-2} \mathbf{O}^\top \mathbf{V}_0^\top \mathbf{X}^\top \mathbf{y} + O_{\mathbb{P}} \bigg(\frac{1}{\sqrt{n}}\bigg) \\ &\xrightarrow{\mathbb{P}} \mathbb{E}[y_i \mathbf{x}_i^\top] \mathbf{V}_0 \mathbf{D}_0^{-2} \mathbf{V}_0^\top \mathbb{E}[\mathbf{x}_i y_i] = C, \end{align*} so we have the result. $\square$