Behavior of a Wishart quadratic form

Question

Let $X \in \mathbb{R}^{n \times d}$ be a random matrix with iid standard Gaussian entries. Let $e_1$ denote the first canonical basis vector in $\mathbb{R}^d$. Define $$ P_d(\lambda) = (1-\lambda) e_1 e_1^T + \lambda \tfrac{1}{d} I_d, \quad \lambda \in (0, 1]. $$ I am trying to understand the behavior of the random variable $$ V_{n, d}(\lambda) = n \, e_1^T\Big(X^T X + P_d(\lambda)^{-1}\Big)^{-1} e_1, \quad \lambda \in (0, 1]. $$ Question: What is the behavior of $F(n, d, \lambda) = \mathbb{E}[V_{n, d}(\lambda)]$? Specifically, fix any $\lambda \in (0, 1]$. Is there a choice of $d = d(n, \lambda)$ such that the limit $$ \lim_{n \to \infty} F\Big(n, d(n,\lambda), \lambda\Big) $$ exists, and is finite and nonzero?

Below I compute a few examples---these are the easy cases---I do not know how to deal with $\lambda \in (0, 1)$ general.

Example 1 ($\lambda = 1$): A classical example is when $\lambda = 1$. In this case, we have $$ F(n, d, 1) = \mathbb{E} ~ \tfrac{1}{d} \mathrm{tr}\Big(\big(\tfrac{1}{n} X^T X + \tfrac{d}{n} I_d\big)^{-1}\Big) $$ Hence if for $\gamma \in (0, \infty)$, we set $d(n, 1) = \gamma n + o(n)$ (for instance, the floor of $\gamma n$), we have $$ F(n, d(n, 1), 1) \to \int \frac{1}{\lambda + \gamma} \, \mathrm{d}\mu^{\sf MP}_\gamma(\lambda), $$ where $\mu^{\sf MP}_\gamma$ denotes the Marchenko-Pastur law, with parameter $\gamma$.

It does not appear that this method works well when $\lambda \neq 1$, since then we have some contribution, in $P_d(\lambda)$, from the matrix $e_1 e_1^T$.

Example 2 ($\lambda = 0$): Note that we have for all $\lambda \in (0, 1]$, $$ V_{n, d}(\lambda) = n e_1^T Q(\lambda)(Q(\lambda) X^T X Q(\lambda) + I)^{-1} Q(\lambda) e_1, $$ where $Q(\lambda) = P(\lambda)^{1/2}$. The advantage of the above representation is that it extends to $\lambda = 0$ naturally, yielding, in distribution $$ V_{n, d}(0) = \Big(\tfrac{1}{n}\chi^2_n + \tfrac{1}{n}\Big)^{-1}. $$ It is easy to see from this that $F(n, d(n, 0), 0) \to 1$, for any choice of $d(n, 0)$.

Answer 1

We can invert $P=P_d(\lambda)$ easily because it is diagonal: $$ P^{-1} = \frac{1}{1-\lambda + \lambda/d} e_1e_1^T + \frac{d}{\lambda} \sum_{j\ge 2} e_j e_j^T. $$ Write $P^{-1}$ as $\frac{d}{\lambda} I_d - y e_1 e_1^T$ and apply the https://en.wikipedia.org/wiki/Sherman%E2%80%93Morrison_formula with $A=X^TX + (d/\lambda) I_d$: $$ (A - ye_1e_1^T)^{-1} = A^{-1} + \frac{y}{1-y e_1^T A^{-1} e_1} A^{-1}e_1e_1^TA^{-1}. $$

Multiplying by $y e_1^T$ on the left and $e_1$ on the right, the right-hand side depends only on $y e_1^T A^{-1} e_1$ which has a finite limit, say $L(\lambda,\gamma)$, under the proportional scaling $d/n \to \gamma$ and the Marchenko-Pastur law.

To see that $y e_1^T A^{-1} e_1$ has a finite limit, observe $$ \text{trace}(A^{-1}) $$ has a limit in probability depending only on $(\lambda,\gamma)$ thanks to the Marcenko-Pastur result. Furthermore, $y/d =\lambda + o(1)$ so that $$ \frac{y}{d}\text{trace}(A^{-1}) $$ also has a finite limit in probability depending only on $(\gamma,\lambda)$.

It remains to link $\text{trace}(A^{-1})$ to $d e_1^T A^{-1} e_1$; we claim that the ratio converges to 1 in probability. To see this, $X=^d XP$ (equality in distribution) for any random rotation $P\in O(d)$ independent of $X$, so that $d e_1^T A^{-1} e_1$ is equal in distribution to $$ d e_1^T (P^T X^T X + P^T (d/\lambda) P)^{-1} e_1 = d v^T (X^TX + (d/\lambda) )^{-1} v = d v^T A^{-1} v $$ where $v = P e_1$. By choosing $P$ distributed according to the Haar measure, $v$ is uniformly distributed on the unit sphere. By concentration of quadratic forms of random vectors uniformly distributed on the sphere (or just Chebyshev's inequality), $$ d v^T A^{-1} v = \text{trace}(A^{-1}) + O_P(1) \|A^{-1}\|_F $$ and the Frobenius norm $\|A^{-1}\|_F$ is of order $1/\sqrt d$. In summary $d e_1^T A^{-1} e_1$ has the same limite as the limit of trace$[A^{-1}]$ given by the Marcenko-Pastur result.

Answer 2

This is a slightly re-written version of jlewk's response (credit to them).

Set $$ y = \frac{d}{\lambda} \frac{d(1-\lambda)}{d(1-\lambda) + \lambda}, \quad \lambda \in (0, 1). $$ Evidently $y/n \to \gamma/\lambda$ if $d/n \to \gamma \in (0, \infty), n \to \infty$.

Using the Sherman-Morrison formula, it can be verified that $$ n V_{n,d}(\lambda) = \frac{W_n(\lambda)}{1 - (y/n) W_n(\lambda)}, \quad W_n(\lambda) := e_1^TQ_n(\lambda)^{-1} e_1, \quad Q_n(\lambda) = \tfrac{1}{n} X^T X + \tfrac{d}{\lambda n} I_d $$ Let $g \in \mathbb{R}^d$ be a standard Gaussian vector. Then in distribution $$ W_n(\lambda) = \frac{A_n(\lambda)}{B_n} \quad A_n(\lambda) = (1/d) g^T Q_n(\lambda)^{-1} g, \quad B_n = \|g\|_2^2/d. $$ By the Hanson-Wright inequality and strong law of large numbers, respectively, it can be shown that $$ A_n(\lambda) - (1/d)\, \mathrm{trace}(Q_n(\lambda)^{-1}) \to 0, \quad \mbox{and} \quad B_n\to 1, $$ in probability and almost surely, respectively, as $d/n \to \gamma \in (0, \infty), n \to \infty$. Therefore, $$ W_n(\lambda) = (1/d) \, \mathrm{trace}(Q_n(\lambda)^{-1}) + o_p(1), \quad n\to\infty $$ and $d/n \to \gamma \in (0, \infty)$. To conclude, we apply the limit relation for $y/n$ and set for $\alpha > 0$, $$ S_\gamma(\alpha) = \int \frac{1}{\lambda + \alpha} \, d\mu^{\sf MP}_\gamma(\lambda), $$ where $\mu^{\sf MP}_\gamma$ denotes the MP law with parameter $\gamma$. Then we clearly have by continuous mapping and the MP theorem that $$ n V_{n, d}(\lambda) \to \frac{S_\gamma(\gamma/\lambda)}{1 - (\gamma/\lambda) S_\gamma(\gamma/\lambda)} $$ in probability as $n \to \infty, d/n \to \gamma \in (0, \infty)$.