Trace of the inverse sample covariance as the number of samples and dimension scale to infinity

Question

Let $x_1,\dots,x_n$ be i.i.d. $N(0,I_{p\times p})$, with $n>p$. Let $\hat S=\frac1n\sum_{i=1}^n x_i x_i^T$ be the sample covariance.

Assume the asymptotic setting where $\frac pn\to \alpha<1$.

Is there a result about the concentration of $\mathbb{E}\left[\operatorname{tr}\left(\hat{S}^{-1}\right)\right]$ in the asymptotic setting? By using Lemma 3.2 in this paper coupled with the Sherman-Morrison formula I believe that $\mathbb{E}\left[\operatorname{tr}\left(\hat{S}^{-1}\right)\right]=\frac{1}{1-\alpha}$, but I haven't been able to make it rigorous.

Are there any existing results about this?

Answer 1

Writing $\hat{S}= \frac1n XX^T$ where $X$ has columns $x_1,\dots,x_n$, we can express $$ \text{tr}(\hat{S}^{-1}) = \sum_{i=1}^p \lambda_i(\hat{S})^{-1} = n\sum_{i=1}^p \sigma_i(X)^{-2} $$ where $\sigma_1(X), \dots, \sigma_p(X)$ are the nonzero singular values of $X$. Then we can use the inverse second moment identity (observed by Tao and Vu here http://arxiv.org/pdf/0807.4898v5.pdf; the rectangular case is given as Lemma 4.14 here: https://arxiv.org/pdf/1109.3343v4.pdf) to express $$\sum_{i=1}^p \sigma_i(X)^{-2}=\sum_{i=1}^p \text{dist}(R_i,R_{-i})^{-2}$$ where $\text{dist}(R_i,R_{-i})$ is the distance from the $i$th row of $X$ to the span of the remaining rows. Since $X$ has normally distributed entries the variables $\text{dist}(R_i,R_{-i})^{2}$ have chi-squared distribution, so the expectation of this sum can be computed explicitly. Unless I have made some mistake it comes out to $\mathbb{E}\frac1p\text{tr}(\hat{S}^{-1}) = \frac{n}{n-p-1} = \frac{1}{1-\alpha}+o(1)$ (note we need the normalization $1/p$).

Answer 2

The formula $\mathbb E[(\sum_{i=1}^n x_ix_i^T)^{-1}] = I_p/(n-p-1)$ is exact. It is known on Wikipedia as the expectation of the inverse wishart distribtion: https://en.wikipedia.org/wiki/Inverse-Wishart_distribution

To prove it, outside of the diagonal for some $i\ne j$: with $X$ the matrix with rows $x_1,...,x_n$ and $e_j$ the $j$-th canonical vector, the fact that $X(I_p - 2e_je_j^T)$ has the same distribution as $X$ (i.e., changing the sign of the $j$-th column) gives that the off-diagional $i,j$-th element is 0.

For the diagonal elements, by symmetry it is enough to compute $trace[(X^TX)^{-1}]$ as asked in the question. By Stein's formula $\mathbb E[x_i^T F(X)]=\sum_j \mathbb E[ e_j^T \frac{\partial}{\partial x_{ij}} F(X)]$ for $F$ valued in $R^p$ we find \begin{align} \mathbb E[x_i^T(X^TX)^{-1}x_i] =\mathbb E\Big[\sum_{j=1}^p e_j^T(X^TX)^{-1} e_j - e_j^T(X^TX)^{-1}(e_j x_i^T + x_i e_j^T)(X^TX)^{-1} x_i\Big]. \end{align} Summing over $i=1,...,n$ $$\mathbb E\sum_{i=1}^n x_i^T(X^TX)^{-1}x_i = n \mathbb E\Big[Tr[(X^TX)^{-1}] - Tr[(X^TX)^{-1}]\sum_{i=1}^n x_i^T(X^TX)^{-1}x_i - Tr[(X^TX)^{-1}\sum_{i=1}^nx_ix_i^T(X^TX)^{-1}] \Big] $$ or equivalently $p=(n-p-1)\mathbb E Tr[(X^TX)^{-1}]$ which is the desired identity.