Let $F\colon\mathbb R^3\to\mathbb R$ be a compactly supported smooth function. I want to compute $$ \int_{\mathbb R^n}\int_{\mathbb R^n} F(\lVert x\rVert^2,\lVert y\rVert^2,\langle x,y\rangle)~\mathrm{d}x~\mathrm{d}y,$$ where $\mathrm{d}x$ is the Lebesgue measure in $\mathbb R^n$, as a three dimensional integral. Doing so requires seems to require computing the volume of certain sphere intersections and does not seem straight-forward. Is there a clever way of doing this? Any references or keywords appreciated.

I'm not sure about a clever way, but the straightforward Calculus 3 type computation is not hard at all. Making the integration in $x$ outer and switching to polar coordinates in $x$, then switching in the inner integral with respect to $y$ to the cylindrical coordinates with the axis of the cylinder parallel to $x$, we get $$ \omega_{n-1}\int_0^\infty r^{n-1}dr\int_{-\infty}^\infty dx\omega_{n-2}\int_0^\infty \rho^{n-2}d\rho F(r^2,xr,x^2+\rho^2) \\ =\omega_{n-1}\omega_{n-2}\iiint r^{n-1}\rho^{n-2}F(x^2,xr,x^2+\rho^2) dr\,dx\,d\rho\,. $$ Now, putting $X=r^2, Y=xr, Z=x^2+\rho^2$, we find $dX=2rdr, dY=xdr+rdx, dZ=2xdx+2\rho d\rho$, so $dXdYdZ=4r^2\rho\,dr\,dx\,d\rho$ whence the integral in question is $$ \frac{\omega_{n-1}\omega_{n-2}}4\iiint (r\rho)^{n-3}F(X,Y,Z)dX\,dY\,dZ $$ It remains to express $r\rho=(XZ-Y^2)^{1/2}$ to get the final answer $$ \frac{\omega_{n-1}\omega_{n-2}}4\iiint (XZ-Y^2)^{\frac{n-3}2}F(X,Y,Z)dX\,dY\,dZ $$ with the domain of integration $X,Z>0, Y^2\le XZ$