2
$\begingroup$

Given a $N \times M$ matrix $X$ comprised of standard normal entries ($M > N$), I'm interested in approximating $E[trace((XX^T\frac{\gamma}{M} + I)^{-1}]$ in terms of $N, M$ and $\gamma$. Unfortunately, I can't necessarily assume $\gamma$ is small. I've had no luck in coming up with any kind of approximation. Thanks!

For context, this problem relates to the effective degrees of freedom for ridge regression.

$\endgroup$

1 Answer 1

4
$\begingroup$

let $\lambda_{1},\lambda_2,\ldots\lambda_N$ be the eigenvalues of $M^{-1}XX^{T}$; including for convenience a factor $1/N$, the quantity you seek is

$$N^{-1}E[{\rm Tr}(XX^{T}\gamma/M)+I)^{-1}]=\int d\lambda \rho(\lambda)(\lambda\gamma+1)^{-1}$$

where $\rho(\lambda)=E[N^{-1}\sum_n\delta(\lambda-\lambda_n)]$ is the eigenvalue density of Wishart matrices; this quantity is known in closed form for $N,M\rightarrow\infty$ at finite ratio $N/M=r\in(0,1]$ (Marchenko-Pastur distribution). I find in this way the answer

$$\lim_{N,M\rightarrow\infty}N^{-1}E[{\rm Tr}(XX^{T}\gamma/M)+I)^{-1}]=(2r\gamma)^{-1}\left(-1-\gamma\sqrt{ab}+\sqrt{(1+a\gamma)(1+b\gamma)}\right)$$

with $a=(1+\sqrt r)^2$ and $b=(1-\sqrt r)^2$. As a check, you can take the limit $\gamma\rightarrow 0$ of this expression and obtain $1$, as it should.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.