Asymptotics of $w^\top G^2 w$, where $w$ is a unit-vector, $G:=X^T(XX^T+t I_n)^{-1}X$, $t > 0$, and $X$ is an $n\times d$ gaussian random matrix

Question

Let $X$ be an random $n \times d$ matrix with entries drawn iid from $N(0,1/d)$ and let $w$ be a unit-vector in $\mathbb R^d$. With $\lambda>0$, and define $G:=X^\top(XX^\top + \lambda I_n)^{-1}X$. Finally, defined $\alpha := w^\top G^2 w$.

Question. In the limit $n,d \to \infty$ with $n/d \to \rho \in (0,\infty)$, what is the limitting value of $\alpha$ as a function of $\lambda$ and $\rho$ ?

A useful subcase is when $\lambda \to 0^+$.

Question. What is the value of $\lim_{\lambda \to 0^+}\lim_{n,d \to \infty \\ n/d \to \rho}\alpha$ as a function of $\rho$ ?

Answer 1

Let me calculate the expectation value of $\alpha$. The probability distribution of $X$ is invariant under orthogonal transformations, so without loss of generality I can orient the unit vector $w$ along one of the axes, $w_i=\delta_{ip}$, $p\in\{1,2,\ldots d\}$. Then $$\mathbb{E}[\alpha]=\mathbb{E}\left(X^T(XX^T+\lambda I)^{-1}XX^T(XX^T+\lambda I)^{-1}X\right)_{pp}.$$ Again because of orthogonal invariance the answer cannot depend on the value of the index $p$, hence we can sum over $p$ and divide by $d$, which gives the trace, $$\mathbb{E}[\alpha]=\frac{1}{d}\mathbb{E}\,{\rm tr}\,\left(X^T(XX^T+\lambda I)^{-1}XX^T(XX^T+\lambda I)^{-1}X\right)$$ $$\qquad=\frac{1}{d}\mathbb{E}\,{\rm tr}\,\frac{W^2}{(W+\lambda I)^2},\;\;W=XX^{T}.$$

For the subcase $\lambda\rightarrow 0$ in the OP we thus find $\mathbb{E}[\alpha]=n/d \to \rho$.

For nonzero $\lambda$ and in the large-$n$ limit the result for $\mathbb{E}[\alpha]$ is the integral of $\mu^2(\mu+\lambda)^{-2}$ weighted by the Marchenko-Pastur distribution $\rho(\mu)$ for the eigenvalues $\mu$ of $W$.