Yes indeed, for fractional $s$ it is related to Modified Bessel Functions of 2nd Kind (Macdonald Function). For $r=|\mathbf{x}|,\ a=|x_{n+1}|$ $$P(r,a)=\frac{a^{2s}}{(r^2+a^2)^{n/2+s}}$$ The Fourier Transform of an n-dimensional radial function $f(r)$ is a radial function in the trasformed space $\mathcal{F}(|\xi|)$. It is given by the following Hankel Transform. $$\mathcal{F}(|\xi|)=\frac{1}{|\xi|^{\frac{n}{2}-1}}\int_0^\infty r^{\frac{n}{2}-1}\left[r\,J_{\frac{n}{2}-1}(|\xi|\cdot r)\right]\cdot f(r)\,dr$$ See, for instance, Sec 11 pg. 63-65 in
Sneddon, Ian N., Fourier transforms, New York: McGraw-Hill Book Company, Inc. (1950). ZBL0038.26801. for a proof.
Also, Sneddon, Ian N., A note on some relations between Fourier and Hankel transforms, Bull. Acad. Pol. Sci., Sér. Sci. Math. Astron. Phys. 9, 799-806 (1961). ZBL0100.31501. Therefore $$\mathcal{F}_P(|\xi|,a)=\frac{a^{2s}}{|\xi|^{\frac{n}{2}-1}}\int_0^\infty J_{\frac{n}{2}-1}(|\xi|\cdot r)\cdot \frac{r^{\frac{n}{2}}\,dr}{(r^2+a^2)^{\frac{n}{2}+s}}$$ But integral is formula 6.565.4 in 7-th Edition of
Gradshteyn, I. S.; Ryzhik, I. M., Table of integrals, series, and products. Ed. by Alan Jeffrey. CD-ROM version 1. 0 for PC, MAC, and UNIX computers., San Diego, CA: Academic Press. (1996). ZBL0918.65001.
It reads (adapted to our notation), for $s>0$ $$\int_0^\infty J_{\frac{n}{2}-1}(|\xi|\cdot r)\cdot \frac{r^{\frac{n}{2}}\,dr}{(r^2+a^2)^{\frac{n}{2}+s}}=\left(\frac{|\xi|}{2}\right)^{\frac{n}{2}+s-1}\cdot\frac{K_s(a\,|\xi|)}{a^s\,\Gamma(\frac{n}{2}+s)}$$ Being $K_s(\cdot)$ the Modified Bessel Function of 2nd Kind. Therefore, $$\mathcal{F}_P(|\xi|,a)=\frac{(a\,|\xi|)^sK_s(a\,|\xi|)}{2^{\frac{n}{2}+s-1}\Gamma(\frac{n}{2}+s)}$$
Since $K_\frac{1}{2}(z)=\sqrt{\frac{\pi}{2z}}\cdot e^{-z}$, for $s=\frac{1}{2}$ we get $$\mathcal{F}_P(|\xi|,a)=\frac{\sqrt{\pi}}{2^\frac{n}{2}\Gamma\left(\frac{n+1}{2}\right)}\cdot e^{-a\,|\xi|}$$ Note: Dimension constant and $2\pi$ in exponent are matter of Fourier Transform's normalization and convention used respectively. By 'Rosetta Table' Formulae 504 at the end of this Wiki page this can be fitted accordingly.
Using question convention -FT unitary, ordinary frequency, $\mathcal{F}^*$- and notation we get,
$$P(|x|,x_{n+1})=\frac{2^{s-\frac{1}{2}}\Gamma\left(\frac{n}{2}+s\right)}{\pi^{\frac{n+1}{2}}}\cdot\frac{x_{n+1}^{2s}}{(|x|^2+|x_{n+1}|^2)^{n/2+s}}$$ producing this amazingly simple expression $$\mathcal{F}_P^*(|\xi|,x_{n+1})=\sqrt{\frac{2}{\pi}}\cdot \left(2\pi\,x_{n+1}\,|\xi|\right)^s\cdot K_s\left(2\pi\, x_{n+1}\,|\xi|\right)$$
This gives $\exp(-2\pi |\xi| x_{n+1})$ for $s=1/2$.
Being so simple, this result should be added as new Formulae 505 in Wiki's Table Fourier Transforms of n-dimensional functions.
I leave this for further reading, a nice article on Fourier Transforms of Fractional Poison Kernels and Fractional Laplacian (See Section 2.8) at
Kwaśnicki, Mateusz, Ten equivalent definitions of the fractional Laplace operator, Fract. Calc. Appl. Anal. 20, No. 1, 7-51 (2017). ZBL1375.47038.