# Integral substitution involving the length and angle of two vectors

Let $$F\colon\mathbb R^3\to\mathbb R$$ be a compactly supported smooth function. I want to compute $$\int_{\mathbb R^n}\int_{\mathbb R^n} F(\lVert x\rVert^2,\lVert y\rVert^2,\langle x,y\rangle)~\mathrm{d}x~\mathrm{d}y,$$ where $$\mathrm{d}x$$ is the Lebesgue measure in $$\mathbb R^n$$, as a three dimensional integral. Doing so requires seems to require computing the volume of certain sphere intersections and does not seem straight-forward. Is there a clever way of doing this? Any references or keywords appreciated.

I'm not sure about a clever way, but the straightforward Calculus 3 type computation is not hard at all. Making the integration in $$x$$ outer and switching to polar coordinates in $$x$$, then switching in the inner integral with respect to $$y$$ to the cylindrical coordinates with the axis of the cylinder parallel to $$x$$, we get $$\omega_{n-1}\int_0^\infty r^{n-1}dr\int_{-\infty}^\infty dx\omega_{n-2}\int_0^\infty \rho^{n-2}d\rho F(r^2,xr,x^2+\rho^2) \\ =\omega_{n-1}\omega_{n-2}\iiint r^{n-1}\rho^{n-2}F(x^2,xr,x^2+\rho^2) dr\,dx\,d\rho\,.$$ Now, putting $$X=r^2, Y=xr, Z=x^2+\rho^2$$, we find $$dX=2rdr, dY=xdr+rdx, dZ=2xdx+2\rho d\rho$$, so $$dXdYdZ=4r^2\rho\,dr\,dx\,d\rho$$ whence the integral in question is $$\frac{\omega_{n-1}\omega_{n-2}}4\iiint (r\rho)^{n-3}F(X,Y,Z)dX\,dY\,dZ$$ It remains to express $$r\rho=(XZ-Y^2)^{1/2}$$ to get the final answer $$\frac{\omega_{n-1}\omega_{n-2}}4\iiint (XZ-Y^2)^{\frac{n-3}2}F(X,Y,Z)dX\,dY\,dZ$$ with the domain of integration $$X,Z>0, Y^2\le XZ$$