That's a good question. This is not quite an answer but a bit long for a comment.

A quick remark is that for $N$ odd a Haar-random element of $O(N)$ can be obtained as $\epsilon U$ where $\epsilon=\pm1$ with equal probability, and $U$ is a Haar-random element of $SO(N)$. So if your monomial has an even number of factors the integrals over $O(N)$ and $SO(N)$ coincide and thus Weingarten calculus is applicable. This is of course because for $N$ odd $-I$ has determinant $-1$ and is in the center of $O(N)$. I don't know if there is a similar trick for $N$ even.

I said it is a good question because, when looking at the vast probability/representation theory literature, I didn't see much as far an analogue of Weingarten calculus for special groups. Even Chatterjee's (and Basu and Ganguly,...) work on $SO(N)$ lattice gauge theories does not seem to use Weingarten calculus. So for $SO(N)$, my answer to the question is: I don't know. However, for $SU(N)$ there is a combinatorial calculus. It is explained in my two answers to

How to constructively/combinatorially prove Schur-Weyl duality?

This technique was worked out explicitly by Creutz but it has its roots in the work of Clebsch and Hilbert in invariant theory. See for example, the averaging operator $[\cdot]$ used by Hilbert on p. 523 of "Ueber die Theorie der algebraischen Formen" is basically the same as Creutz's formula for $SU(2)$. Also note that if a combinatorial Weingarten-like calculus for $SO(N)$ is perhaps missing, there is at least an Euler angle parametrization due to Hurwitz (see this review by Diaconis and Forrester).

That's a good question. This is not quite an answer but a bit long for a comment.

A quick remark is that for $N$ odd a Haar-random element of $O(N)$ can be obtained as $\epsilon U$ where $\epsilon=\pm1$ with equal probability, and $U$ is a Haar-random element of $SO(N)$. So if your monomial has an even number of factors the integrals over $O(N)$ and $SO(N)$ coincide and thus Weingarten calculus is applicable. This is of course because for $N$ odd $-I$ has determinant $-1$ and is in the center of $O(N)$. I don't know if there is a similar trick for $N$ even.

I said it is a good question because, when looking at the vast probability/representation theory literature, I didn't see much as far an analogue of Weingarten calculus for special groups. Even the work of Chatterjee (and Basu and Ganguly,...) on $SO(N)$ lattice gauge theories does not seem to use Weingarten calculus. So for $SO(N)$, my answer to the question is: I don't know. However, for $SU(N)$ there is a combinatorial calculus. It is explained in my two answers to

How to constructively/combinatorially prove Schur-Weyl duality?

This technique was worked out explicitly by Creutz but it has its roots in the work of Clebsch and Hilbert in invariant theory. See for example, the averaging operator $[\cdot]$ used by Hilbert on p. 523 of "Ueber die Theorie der algebraischen Formen" is basically the same as Creutz's formula for $SU(2)$. Also note that if a combinatorial Weingarten-like calculus for $SO(N)$ is perhaps missing, there is at least an Euler angle parametrization due to Hurwitz (see this review by Diaconis and Forrester).