The Poisson kernel for the orthogonal group was calculated by Benjamin Béri in Generalization of the Poisson kernel to the superconducting random-matrix ensembles. This is in the context where $X$ is the scattering matrix of a superconductor in the Cartan symmetry class D (broken time-reversal and spin-rotation symmetries).
The result for $X\in{\rm SO}(N)$ (real unitary matrices with determinant $+1$) is
$$P_{\rm SO}(X)\propto\frac{1}{|\operatorname{det}(X-Z)|^{N-1}},$$
to be contrasted with
$$P_{\rm U}(X)\propto\frac{1}{|\operatorname{det}(X-Z)|^{2N}}$$
for $X\in{\rm U}(N)$.
Since the exponent $N-1$ differs from the expectation in the OP, let me check the reproducing property for ${\rm SO}(2)$ and $Z=zI_2$ (with $-1<z<1$). In that case the Poisson kernel is
$$P_{\rm SO}(X)d\mu(X)=f(\theta)\,d\theta,\;\;f(\theta)=\frac{1-z^2}{2 \pi \left(z^2-2 z \cos \theta+1\right)},
$$
$$\text{in the parameterization}\;\;X(\theta)=\begin{pmatrix}\cos\theta&\sin\theta\\
-\sin\theta&\cos\theta\end{pmatrix}\;\;0<\theta<2\pi,$$
and one readily checks that
$$\langle X^p\rangle\equiv\int_0^{2\pi} X^p(\theta)f(\theta)\,d\theta=z^p I_2=Z^p.$$
The reproducing property for the orthogonal Poisson kernel fails if $Z$ is not proportional to the unit matrix, for example, if $Z={{z\;0}\choose{0\,-z}}$ one has $\operatorname{det}(X-Z)=1-z^2$, independent of $X$ and $\langle X\rangle=0\neq Z$.
To generalize the Poisson kernel to all three classical groups ${\rm U}(N)$, ${\rm SO}(N)$, ${\rm Sp}(N)$, one defines an $N\times N$ subunitary matrix $Z$ as the submatrix of the $2N\times 2N$ unitary matrix
$$\Omega=\begin{pmatrix}Z&T'\\
T&Z'\end{pmatrix}.$$
The $N\times N$ unitary matrix $X$ is then constructed by
$$X=Z+T'X_0(1-Z'X_0)^{-1}T,$$ with $X_0$ an $N\times N$ unitary matrix distributed according to the Haar measure on the unitary, orthogonal, or symplectic group. The Poisson kernel is then defined as the corresponding probability distribution function $P(X)$ for a given $Z$. The result is $P(X)\propto|\operatorname{det}(X-Z)|^{-\beta N+\gamma}$, with
$\beta=2,\gamma=0$ for ${\rm U}(N)$, $\beta=1,\gamma=1$ for ${\rm SO}(N)$, and $\beta=1,\gamma=-1$ for ${\rm Sp}(N)$.
$$\mbox{}$$
The reproducing property, $\int P(X)f(X)d\mu(X)=f(Z)$, is ensured if the average of the matrix product $X_0(Z'X_0)^p$ vanishes for all $p=0,1,2\ldots$, which happens for any integer $N$ and any $Z$ in the unitary group, and for even $N$ and $Z\propto I_N$ in the orthogonal or symplectic groups.