Let $g:\mathbb R \to \mathbb R $ be a continuous function which is

- "sufficiently smooth" (e.g $\mathcal C^3$) around $0$, and
- "sufficiently integrable" (e.g integrable w.r.t $N(0,1)$).

Let $d'$ and $d$ be positive integers tending to infinity such that $d'/d \to \rho \in (0,\infty)$.

Let $X$ be an $d' \times d$ random matrix with iid rows $x_1,\ldots,x_D$ from $N(0,(1/d)I_d)$, let $\widetilde{X}$ be the $d'\times d$ random matrix with rows $\widetilde{x}_i := x_i/\|x_i\|$. Thus each $\widetilde{x}_i$ is uniformly distributed on the unit-sphere in $\mathbb R^{d}$.

For $x \sim N(0,(1/d)I_d)$ independent of $X$, define an $d' \times d'$ random psd matrices $B$ and $\widetilde{B}$ by $$ \begin{split} b_{ij} &:= (x_i^\top x_j)\mathbb E_x[g(x^\top x_i)g(x^\top x_j)],\\ \widetilde{b}_{ij} &:= (\widetilde{x}_i^\top \widetilde{x}_j)\mathbb E_x[g(\widetilde{x}^\top \widetilde{x}_i)g(\widetilde{x}^\top \widetilde{x}_j)]. \end{split} $$

Question.Are the following true ?$$ \begin{split} \|B-g(0)^2 XX^\top\|_{op} &\overset{p}{\to} 0,\\ \|\widetilde{B}-g(0)^2 \widetilde{X}\widetilde{X}^\top\|_{op} &\overset{p}{\to} 0. \end{split} $$

## Empirical evidence

Here are results for some experiments with $d'=200$ and $d=300$, and different choices for $g$, both smooth and rough. Judgin from these observations, it seems the above question is affirmative!

## An sloppy attempt

Let us give a heuristic (and incorrect!) argument why one would expect $$ \|\widetilde{B}-g(0)^2\widetilde{X}\widetilde{X}^\top\|_{op} \overset{p}{\to} 0 $$ to hold.

For simplicity of notation, let $w_i:=\widetilde{x}_i$ henceforth. Let the matrix $\widetilde{U}$ be the $n \times n$ random psd matrix be defined by $\widetilde{u}_{ij} := \mathbb E_z[g(z^\top w_i)g(z^\top w_j)]$, where $z=(z_1,\ldots,z_d)$ is uniform on the unit-sphere on $\mathbb R^d$, and independent of the $w_i$'s.

Note that $\widetilde{B}$ the Hadamard product of $\widetilde{U}$ with $\widetilde{X}\widetilde{X}^\top$.

Now, due to rotational-invariance, we may write $\widetilde{u}_{ij} = u_d(z_i^\top z_j)$, where $u_d:[-1,1] \to \mathbb R$ is defined by $$ u_d(t) := \mathbb E_z[g(z_1)g(tz_1+(1-t^2)^{1/2} z_2)]. $$ Thus, $\widetilde{B}$ is a dot-product kernel matrix, via an envelop function $u_d$ which varies with the dimension $d$. This dependence on $d$ is bad news for us.

Let us ignore the dependence of $u_d$ on $d$. Of course, this isincorrect(and will be the only sloppy part of our arguments), but let's do it anyways.

We can then invoke Theorem 2.3 of El Karoui '10 to get that $\|\widetilde{U}-A\|_{op} \overset{p}{\to} 0$, where $$ A := u_d(0) 1_{d'}1_{d'}^\top + u_d'(0)\widetilde{X}\widetilde{X}^\top + \gamma I_{d'}, $$ with $\gamma := u_d(1)-u_d(0)-u_d'(0)$.

Finally, using exponential-in-$d$ concentration of $z_1^2 + z_2^2$, it is to see that

Fact.If $G:\mathbb R^2 \to \mathbb R$ is a continuous function, then $\mathbb E_z[G(z_1,z_2)] \to G(0,0)$.

We deduce that if our $g$ is $\mathcal C^1$ at $0$, then

$$ \begin{split} u_d(1) &= \mathbb E_z[g(z_1)^2] \to g(0)^2,\\ u_d(0) &=\mathbb E_z[g(z_1)g(z_2)] \to g(0)^2,\\ u_d'(0) &= \mathbb E_z[z_1 g(z_1)g'(z_2)] \to 0,\\ \gamma &= u_d(1)-u_d(0)-u_d'(0) \to 0, \end{split} $$

from which it would follow that $\|\widetilde{U}-g(0)^2 1_{d'}1_{d'}\|_{op} \overset{p}{\to} 0$, and so $\|\widetilde{B}-g(0)^2 \widetilde{X}\widetilde{X}^\top\|_{op} \overset{p}{\to} 0$, as claimed.