Asymptotic property of the left singular vectors of i.i.d. data matrix Let $\mathbf{X}$ be $(n \times p)$-dimensional data matrix ($n > p$) whose rows $\mathbf{x}_i$ are i.i.d. with some finite moments:
$$
   \mathbf{X}^\top = [\mathbf{x}_1, \ldots \mathbf{x}_n]^\top.
$$
By the singular decomposition, we can obtain
$$
   \mathbf{X} = \mathbf{U} \mathbf{D} \mathbf{V}^\top = \mathbf{U}_{:r} \mathbf{D}_{:r} \mathbf{V}_{:r}^\top,
$$
where $r$ is the rank of $\mathbf{X}$ which may $r<p$. Notice that here I used a Matlab colon notation ":" to denote submatrices
\begin{align*}
   \mathbf{U}_{:r} &:= [\mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_r],  \\
\mathbf{D}_{:r} &:= \operatorname{diag}([d_1,d_2,\ldots,d_r]), \\
\mathbf{V}_{:r} &:= [\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_r],
\end{align*}
(Notice that $\mathbf{U}_{:r}$ is not a square matrix, so $\mathbf{U}_{:r}^\top \mathbf{U}_{:r} = \mathbf{I}$, but $\mathbf{U}_{:r}\mathbf{U}_{:r}^\top \neq \mathbf{I}$). Then the question is

Question.
As $n \to \infty$ while $p$ is fixed, what can we know about the asymptotic property of $\mathbf{U}_{:r}$, for example, the limit of $n^{-1} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y}$, where $\mathbf{y}$ is a random vector whose elements are i.i.d. and independent to $\mathbf{X}$?

As an example, we can know about the asymptotic properties of $\mathbf{D}_{:r}$ and $\mathbf{V}_{:r}$: by the law of large numbers, as $n \to \infty$ (while $p$ is fixed),
$$
   \frac{1}{n} \mathbf{X}^\top \mathbf{X} = \frac{1}{n} \mathbf{V}_{:r} \mathbf{D}_{:r}^2 \mathbf{V}_{:r}^\top \xrightarrow{\mathbb{P}} \mathbb{E} \big[ \mathbf{x}_i \mathbf{x}_i^\top \big],
$$
so $\mathbf{V}_{:r}$ and $\mathbf{D}_{:r}^2$ converges to the eigenvectors and eigenvalues of $\mathbb{E} \big[ \mathbf{x}_i \mathbf{x}_i^\top \big]$.
Of course that since the number of rows of $\mathbf{U}$ goes to infinity, we cannot directly say what is the limit of $\mathbf{U}_{:r}$, but we may know (or at least the existence) the limit of $n^{-1} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y}$.
Here we assume that $\mathbf{X}$ and $\mathbf{y}$ have finite moments (and sub-Gaussianity).
If this question is too elementary, I really apologize for that, but this question is very important to prove the asymptotic properties of my statistical estimator. It would be really appreciated if you give any help.
Thanks,
 A: This is for the self-reference.

Claim.
There exists some $C>0$ such that
$$\frac{1}{n} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y} \xrightarrow{\mathbb{P}} C.$$

Proof.
The main idea of the proof is that linear approximation of the left singular vectors $\mathbf{U}_{:r}$ using the limit of $\mathbf{V}_{:r}$ and $\mathbf{D}_{:r}$.
(Step 1: Limit of $\mathbf{V}_{:r}$ and $\mathbf{D}_{:r}$). Since we know that $n^{-1} \mathbf{X}^\top \mathbf{X} \xrightarrow{\mathbb{P}} \mathbb{E}[\mathbf{x}_i \mathbf{x}_i^\top]$,by the Davis-Kahan theorem and Weyl's inequality, we have
\begin{align*}
  &\mathbf{V}_{:r} = \mathbf{V}_0 \mathbf{O}^\top + O_{\mathbb{P}}\bigg(\frac{1}{\sqrt{n}}\bigg), &\frac{\mathbf{D}_{:r}^2}{n} = \frac{\mathbf{D}_{0}^2}{n} + O_{\mathbb{P}}\bigg(\frac{1}{\sqrt{n}}\bigg),
\end{align*}
where $\mathbf{V}_0$ and $\mathbf{D}_0^2/n$ are the first $r$ eigenvectors and eigenvalues of $\mathbb{E}[\mathbf{x}_i\mathbf{x}_i^\top]$, respectively, and $\mathbf{O}$ is some orthogonal matrix which depends on $\mathbf{V}_{:r}$ and $\mathbf{V}_0$.
(Step 2: Approximation of $\mathbf{U}_{:r}$).
By using the previous result, we can approximate the left singular vectors $\mathbf{U}_{:r}$ as follows:
\begin{align*}
  \mathbf{U}_{:r} = \mathbf{X} \mathbf{V}_{:r} \mathbf{D}_{:r}^{-1} = \mathbf{X} \mathbf{V}_0 \mathbf{O}^\top \mathbf{D}_0^{-1} + \mathbf{X} \mathbf{R} \approx \mathbf{X} \mathbf{V}_0 \mathbf{O}^\top \mathbf{D}_0^{-1},
\end{align*}
where
\begin{align*}
  \mathbf{R} &:= \mathbf{V}_{:r} \mathbf{D}_{:r}^{-1} - \mathbf{V}_0 \mathbf{O}^\top \mathbf{D}_0^{-1}  \\
  &= (\mathbf{V}_{:r} - \mathbf{V}_0 \mathbf{O}^\top) \mathbf{D}_{:r}^{-1} + \mathbf{V}_0 \mathbf{O}^\top (\mathbf{D}_{:r}^{-1} - \mathbf{D}_0^{-1}) \\
&= O_{\mathbb{P}}\bigg(\frac{1}{n}\bigg).
\end{align*}
Therefore, we have
\begin{align*}
  \frac{1}{n} \mathbf{y}^\top \mathbf{U}_{:r} \mathbf{U}_{:r}^\top \mathbf{y} &= \frac{1}{n} \mathbf{y}^\top \mathbf{X} \mathbf{V}_0 \mathbf{O} \mathbf{D}_0^{-2} \mathbf{O}^\top \mathbf{V}_0^\top \mathbf{X}^\top \mathbf{y} + O_{\mathbb{P}} \bigg(\frac{1}{\sqrt{n}}\bigg) + O_{\mathbb{P}} \bigg(\frac{1}{n}\bigg) \\
&= \frac{1}{n} \mathbf{y}^\top \mathbf{X} \mathbf{V}_0 \mathbf{D}_0^{-2} \mathbf{O}^\top \mathbf{V}_0^\top \mathbf{X}^\top \mathbf{y} + O_{\mathbb{P}} \bigg(\frac{1}{\sqrt{n}}\bigg)  \\
&\xrightarrow{\mathbb{P}} \mathbb{E}[y_i \mathbf{x}_i^\top] \mathbf{V}_0 \mathbf{D}_0^{-2} \mathbf{V}_0^\top \mathbb{E}[\mathbf{x}_i y_i] = C,
\end{align*}
so we have the result.     $\square$
