This is not a full answer, just an outline. It may require additional regularity assumptions on $f$ and $g$. $\newcommand{\bR}{\mathbb{R}}$ $\DeclareMathOperator{\SO}{SO}$ For $x\in\bR^{n+1}\setminus 0$ we set $$\bar{x}:=\frac{1}{|x|}x.$$
I assume that $xy$ denotes the inner product. Note that
$$
F[f](x)=\int_0^\infty\left(\int_{S^n} f(y)g(|xy|/t)dy\right) t^{-n-1}dt
$$
$$
= \int_0^\infty\left(\int_{S^n} f(y)g(|x| |\bar{x} y|/t)dy\right) t^{-n-1}dt
$$
($t=s|x|$)
$$
= |x|^{-n}\int_0^\infty\left(\int_{S^n} f(y)g(|\bar{x} y|/s)dy\right) s^{-n-1}ds=|x|^{-n}F(\bar{x}).
$$
For $s>0$ define $\newcommand{\eT}{\mathscr{T}}$
$$
\eT_s:L^2(S^n)\to L^2(S^n),\;\;\eT_s[f](x)=\int_{S^n}f(x)g(|xy|/s) dy,\;\;\forall x\in S^{n}.
$$
(This requires some assumption on $g$.) Observe next that we have a right action of $\SO(n+1)$ on $L^2(S^n)$. For $A\in\SO(n+1)$ define
$$
L^2(S^n)\ni f\mapsto A^*f\in L^2(S^n),\;\;A^*f(x)=f(Ax).
$$
Note that
$$
\eT_s[A^*f](x) = \int_{S^n}f(Ax)g(|Axy|/s) dy= \int_{S^n}f(Ax)g(|AxAy|/s) dy
$$
$$
= \int_{S^n}f(Ax)g(|xy|/s) dy=\eT_s[f](Ax)
$$
so that
$$\eT_s[A^*f]=A^*\eT_s[f].
$$
In other words, the transformation $\eT_s$ is equivariant with respect to the action of $\SO(n+1)$ and thus, according to Schur's Lemma, it acts as multiplication by constants on the irreducible components of this $\SO(n+1)$ representation on $L^2(S^n)$.
These are the spaces of homogeneous harmonic polynomials or, equivalently, the eigenspaces of the Laplacian on the round $n$-dimensional sphere. As such they coincides with the restrictions to the sphere of homogeneous harmonic polynomilas.
Denote by $\newcommand{\bH}{\mathbb{H}}$ $\bH_d$ space of (restrictions to $S^n$) of homogeneous polynomials of degree $d$ on $\bR^{n+1}$. Thus, $\forall s>0$, $d>0$ there exists a constant $c_d(s)$ such that
$$
\eT_s[P]=c_d(s)P,\;\;\forall P\in \bH_d.
$$
Let me explain how to find this constant. Denote by $\newcommand{\bx}{\boldsymbol{x}}$ $\bx^+=(1,0,0,\dotsc,0)\in\bR^{n+1}$ the North Pole in $S^n$ and choose $P\in\bH_d$ such that $P(\bx^+)=1$. Then
$$
\eT_s[P](\bx^+)=c_d(s)P(\bx^+)=c_d(s).
$$
Hence
$$
c_d(s)=\int_{S^n}P(y)g(|\bx^+y|/s) dy.
$$
Fortunately the spaces $\bH_d$ are well understood and the above integral can be explicitly described as a $1$-dimensional integral involving $g$ and Legendre polynomials. This is the so called Funk-Hecke formula ; see Sec. 1.4 of
C. Muller: Analysis of Spherical Symmetries in Euclidean Spaces, Springer Verlag, 1998.
Now observe that if $f\in\bH_d$ and $x\in S^n$ then
$$
F[f](x)=\int_0^\infty \eT_s[P] s^{-n-1} ds=\left(\int_0^\infty c_d(s) s^{-n-1} ds\right)P.
$$
Thus, everything boils down to computing Fourier transforms of homogeneous functions of the form,
$$\frac{1}{|x|^{n+d}}P_d(x), $$
where $P_d$ is a homogenous harmonic polynomial of even degree $d$ in $n+1$ variables.