It is true under the assumption $k,d\to+\infty$ while $k/d\to 0$, in the sense that there exists a matrix $G\in R^{d\times k}$ with iid $N(0,1)$ entries such that the pseudo inverse $R^+$ satisfies $$ \|R^+ - G\|_{op} /\|G\|_{op} \to^P 0. $$ Above, the denominator $\|G\|_{op}$ could be replaced by $\sqrt d$ since $\|G\|_{op}/\sqrt{d}\to^P 1$ because, e.g., of some well known concentration inequalities for the smallest and largest singular values of $G$, namely $$ P( \sqrt{d} - \sqrt{k} - t \le s_{\min}(G) \le s_{\max}(G) = \|G\|_{op} \le \sqrt d + \sqrt k + t) \le 2e^{-t^2/2}. $$ (One can for instance use $t=\sqrt{\log(d)}$ to obtain vanising probabilities).

Let us now explain why $\|R^+ - G\|_{op} /\sqrt{d} \to^P 0$ holds.

If $R$ has iid entries with $k$ rows and $d$ columns ($k<d$), then by rotational invariance the SVD $R=UDV^T$ satisfies

• $U\in O(k)$ is uniformly distributed (Haar measure on $O(k)$,
• the diagonal matrix $D\in R^k$ contains the random singularvalues,
• $V\in R^{d\times k}$ has $k$ orthonormal columns and is distributed according to the Grassmanian.

And $(U, D, V)$ are independent.

The pseudo-inverse is $R^+ = VD^{-1} U^T$. Now, let $\tilde D$ be an independent copy of $D$, independent of everything else mentioned so far. Then $G=V\tilde D U^T$ is equal in distribution to a matrix with iid $N(0,1)$ entries because $V, \tilde D, U$ are independent and as in the bullet points above. Because of the concentration of the smallest and largest eigenvalues, $\|D^{-1}/\sqrt d -I_k\|_{op}\to^P0$ and $\|\tilde D/\sqrt d - I_k\|_{op}\to^P 0$ so that $\|D^{-1} - \tilde D\|_{op}/\sqrt{d} \to^P 0$. It follows that $\|G-R^+\|_{op}/\sqrt d=\|V(D^{-1} - \tilde D)U^T\|_{op}/\sqrt d \to^P 0$ as claimed.

It is true under the assumption $k,d\to+\infty$ while $k/d\to 0$, in the sense that $R^T\in R^{d\times k}$ has obviously iid $N(0,1)$ entries, and satisfies $$ \|\sqrt d R^+ - R^T/\sqrt d\|_{op} \to^P 0. $$ First let us recall some well known concentration inequalities for the smallest and largest singular values of $R$, namely $$ P( \sqrt{d} - \sqrt{k} - t \le s_{\min}(R) \le s_{\max}(R) = \|R\|_{op} \le \sqrt d + \sqrt k + t) \le 2e^{-t^2/2}. $$ if $k/d\to 0$, one can for instance use $t=\sqrt{\log(d)}$ to obtain that $s_{\min}(R)/\sqrt d\to^P 1$ and similarly $\|R\|_{op}/\sqrt d\to ^P 1$.

Let us now explain why $\|\sqrt d R^+ - R^T/\sqrt d\|_{op} \to^P 0$. Consider the SVD $R=UDV^T$. Then the pseudo-inverse is $R^+ = VD^{=1} U^T$ and \begin{align*} \|\sqrt d R^+ - R^T/\sqrt d\|_{op} &=\|U(\sqrt d D^{-1} - d^{-1/2} D)V\|_{op} \\&=\|\sqrt d D^{-1} - d^{-1/2} D\|_{op} \\&\le \|\sqrt d D^{-1} -I_k\|_{op} + \|I_k - d^{-1/2} D\|_{op}. \end{align*} If $k/d\to 0$, the right-hand side converges to 0 in an event of probability at least $1-2/p$ by taking $t=\sqrt{\log d}$ in the above concentration inequality.

If $R$ has iid entries with $k$ rows and $d$ columns ($k<d$), then by rotational invariance the SVD $R=UDV^T$ satisfies

• $U\in O(k)$ is uniformly distributed (Haar measure on $O(k)$,
• the diagonal matrix $D\in R^k$ contains the random singularvalues,
• $V\in R^{d\times k}$ has $k$ orthonormal columns and is distributed according to the Grassmanian.

And $(U, D, V)$ are independent.

The pseudo-inverse is $R^+ = VD^{-1} U^T$. Looking at one entry specifically, say we multiply to the left by $x\in S^{d-1}$ and to the right by $y\in S^{k-1}$, then $\hat y = U y \in S^{k-1}$ is uniformly distributed on the sphere and so is $\hat x = V^Tx \in S^{k-1}$, and the random variable of interest is $Z = \hat x^T (D^{-1}) \hat y.$ Writing $\hat x = g/\|g\|$ for standard normal $g$, $\|g\|^2\sim \chi^2_k$ and the above random variable is equal in distirbution to $$ N(0,1) \cdot \|D^{-1} \hat y\|/ \sqrt{\chi^2_k} $$ If $d,k$ are such that $\|D^{-1} \hat y\|^2\approx {trace[D^{-2}]}/k$ by, e.g., the Hanson-Wright concentration inequality, and if $trace[D^{-2}]=trace[(RR^T)^{-1}]$ converges to its expectation $\frac{k}{d-k-1}$ (https://en.wikipedia.org/wiki/Inverse-Wishart_distribution), the asymptotic variance can be characterized.

It is true under the assumption $k,d\to+\infty$ while $k/d\to 0$, in the sense that there exists a matrix $G\in R^{d\times k}$ with iid $N(0,1)$ entries such that the pseudo inverse $R^+$ satisfies $$ \|R^+ - G\|_{op} /\|G\|_{op} \to^P 0. $$ Above, the denominator $\|G\|_{op}$ could be replaced by $\sqrt d$ since $\|G\|_{op}/\sqrt{d}\to^P 1$ because, e.g., of some well known concentration inequalities for the smallest and largest singular values of $G$, namely $$ P( \sqrt{d} - \sqrt{k} - t \le s_{\min}(G) \le s_{\max}(G) = \|G\|_{op} \le \sqrt d + \sqrt k + t) \le 2e^{-t^2/2}. $$ (One can for instance use $t=\sqrt{\log(d)}$ to obtain vanising probabilities).

Let us now explain why $\|R^+ - G\|_{op} /\sqrt{d} \to^P 0$ holds.

If $R$ has iid entries with $k$ rows and $d$ columns ($k<d$), then by rotational invariance the SVD $R=UDV^T$ satisfies

• $U\in O(k)$ is uniformly distributed (Haar measure on $O(k)$,
• the diagonal matrix $D\in R^k$ contains the random singularvalues,
• $V\in R^{d\times k}$ has $k$ orthonormal columns and is distributed according to the Grassmanian.

And $(U, D, V)$ are independent.

The pseudo-inverse is $R^+ = VD^{-1} U^T$. Now, let $\tilde D$ be an independent copy of $D$, independent of everything else mentioned so far. Then $G=V\tilde D U^T$ is equal in distribution to a matrix with iid $N(0,1)$ entries because $V, \tilde D, U$ are independent and as in the bullet points above. Because of the concentration of the smallest and largest eigenvalues, $\|D^{-1}/\sqrt d -I_k\|_{op}\to^P0$ and $\|\tilde D/\sqrt d - I_k\|_{op}\to^P 0$ so that $\|D^{-1} - \tilde D\|_{op}/\sqrt{d} \to^P 0$. It follows that $\|G-R^+\|_{op}/\sqrt d=\|V(D^{-1} - \tilde D)U^T\|_{op}/\sqrt d \to^P 0$ as claimed.