Let $n,d \to \infty$ with $n/d \to \gamma \in (0,\infty)$. Let $X$ be a random $n \times d$ matrix independent rows uniformly distributed on the the unit-sphere in $\mathbb R^d$ and let $y$ be a random vector in $\mathbb R^n$ independent of $X$, with iid entries uniformly distributed on $\{\pm 1\}$. If it helps, I'm fine with instead assuming that $y_1,\ldots,y_n \sim N(0,1)$.

Now, for $\lambda \ge 0$ let $Q_\lambda(\omega) := (\omega + \lambda I_n)^{-1}$ and set $\beta_\lambda := X^\top Q_\lambda(XX^\top) y \in \mathbb R^d$, the unique solution to the (random) ridge-regression problem

$$ \arg\min_{\beta \in \mathbb R^d}\frac{1}{2}\|y-X\beta\|^2 + \lambda \|\beta\|^2. $$

Question.What is a good asymptotic lower-bound for $\|\beta_\lambda\|^2$ (valid with high probability or almost-surely) ?

**N.B.:** I'm particularly interested in the "ridgeless" limit $\lambda \to 0^+$.

## Attempt 1

Note that $\beta_\lambda$ has centered multivariate distribytion with covariance matrix $R_\lambda(XX^T)$, where $ R_\lambda(\omega) := Q_\lambda(\omega)\omega Q_\lambda(\omega), $ and so one may write $$ \dfrac{\|\beta_\lambda\|^2}{d} \to \dfrac{1}{d}\mbox{Tr}(R_\lambda(XX^\top)) = \mbox{tr}_d(R_\lambda(XX^\top)), $$ where $\mbox{tr} = (1/d)\mbox{Tr}$ is a normalized trace operator. Since $XX^\top$ is a "free random variable" (in the sense of free probability), and $R_\lambda(XX^\top)$ is a rational expression in $\omega=XX^\top$, one would expect $\|\beta_\lambda\|^2$ to have a complete analytic description via free probability.