Disclaimer. This post is just to further simplify @Carlo Beenakker's answer and highlight some potential benefits. It would be a very long comment, so I decided to post it here instead.
With an obvious abuse of notation, let us write $H_n:=H_n(0)$, the $n$th Hermite number. For even $n$, one has
$$
\Gamma(1/2-n/2) = \frac{(-4)^{n/2}(n/2)!\sqrt{\pi}}{n!} = \frac{2^{n/2}2^{n/2}(-1)^{n/2}(n/2)!\sqrt{\pi}}{n!} = \frac{2^{n/2}\sqrt{\pi}}{H_n},
$$
and so we deduce that $\dfrac{\pi2^{(m+n)/2}}{\Gamma(1/2-m/2)\Gamma(1/2-n/2)} = H_nH_m$, and
$$
\begin{split}
\dfrac{\pi2^{(m+n)/2}}{\Gamma(-m/2)\Gamma(-n/2)} &= \dfrac{\pi2^{(m+n)/2}}{\Gamma(1/2-(m+1)/2)\Gamma(1/2-(n+1)/2)}=(1/4)H_{m+1}H_{n+1}
\end{split}
$$
Thus, we get the following instructive formula
$$
s_{nm} \approx \begin{cases}H_nH_m+(1/d)\left(H_nH_{m+1} + H_{n+1}H_m\right),&\mbox{ if }n,m\text{ even},\\
(1/d)H_{n+1}H_{m+1}t,&\mbox{ else,}
\end{cases}
\tag{1}
$$
where we have used the fact that $nH_n = -H_{n+1}$ for every integer $n \ge 0$.
Application
To see the importance of rewriting @Carlo's formula in the form (1), consider the following claim (which settles another question here Approximate the singular values of a certain random dot-product kernel matrix (in the sense of El Karoui, Cheng-Singer, etc.))
Claim. If $g$ is twice continuously-differentiable on $(-1,1)$, then
$$
\mathbb E[g'(X^\top a)g'(X^\top b)]=g'(0)^2+\mathcal O(1/d)+\mathcal O(1/d)t.
$$
It should be noted that the above estimate has been obtained here https://mathoverflow.net/a/405773/78539, under the much more restrictive condition that $g$ is $\mathcal C^2$ on $(-1,1)$ and $\mathcal C^6$ at $0$.
Proof of Claim. Under the hypothesis, $g'$ has a pointwise convergence Hermite expansion (thanks to this post https://mathoverflow.net/a/145235/78539)
$$
g'(x) = \sum_{n \ge 0} b_n(g') H_n(x),\,\forall x \in (-1,1).
$$
In particular, $g'(0) = \sum_{n \ge 0\text{ even }}b_n(g') H_n(0)$.
Here, $b_n(g') := \mathbb E_{z \sim N(0,1)}[g(z)H_n(z)]$ is the $n$th Hermite coefficient of $g$. Recall the important formula
$$
b_n(g') = b_{n+1}(g),\,\forall n \ge 0.
\tag{2}
$$
Now, one has
$$
\begin{split}
\mathbb E[g'(X^\top a)g'(X^\top b)] &= \sum_{n \ge 0}\sum_{m \ge 0} b_n(g')b_m(g')\mathbb E[H_n(X^\top a)H_m(X^\top b)]\\
&= \sum_{n}\sum_{m} b_n(g')b_m(g')s_{n,m}\\
&\overset{(1)}{=} \sum_{n,m \ge 0\text{ even }}b_n(g')H_n(0)b_m(g')H_m(0)\\
&\quad+ \mathcal O(1/d)\sum_{n,m}b_n(g')H_nb_m(g')H_{m+1} +b_n(g')H_{n+1}H_m\\
&\quad+ \mathcal O(1/d)t\sum_{n,m \ge 1\text{ odd }}b_n(g')H_{n+1}b_m(g')H_{m+1}\\
&\overset{(2)}{=} \left(\sum_{n \ge 0}b_n(g')H_n\right)^2\\
&\,+ \mathcal O(1/d)\sum_{n,m}b_n(g')H_nb_{m+1}(g)H_{m+1} +b_{n+1}(g)H_{n+1}H_m\\
&\,+ \mathcal O(1/d)t\sum_{n,m \ge 1\text{ odd }}b_{n+1}(g)H_{n+1}b_{m+1}(g)H_{m+1}\\
&= g'(0)^2 + \mathcal O(1/d)g'(0)(g(0)-b_0(g)) + \mathcal O(1/d)t(g(0)-b_0(g))^2\\
&= g'(0)^2 + \mathcal O(1/d) + \mathcal O(1/d) t,
\end{split}
$$
as claimed.