Here is a more detailed version of my sketch in the comments above.
First of all, let $\rho$ be a compactly supported smooth function (cutoff function) such that $\rho=1$ on $B_1$. Write $f=\rho f+(1-\rho)f$. Since $\rho f$ is compactly supported, it is $L^1$, and your old argument applies to it, so it remains to bound that for $(1-\rho)f$, whose support is disjoint from $B_1$, which we simply denote as $f$ henceforth.
Then we do the $s$ integral first. Integration by parts once gives
$$ \int_0^\infty e^{-s}e^{i(s+|x|)^2/t}ds=\frac{te^{i|x|^2/t}}{2i|x|}-\int_0^\infty t\frac{d}{ds}(\frac{e^{-s}}{2i(s+|x|)})e^{i(s+|x|)^2/t}ds. $$
Integrate by parts a second time gives
$$ \int_0^\infty e^{-s}e^{i(s+|x|)^2/t}ds
=\frac{te^{i|x|^2/t}}{2i|x|}+\frac{t}{2i|x|}(\frac{t}{2i|x|}+\frac{t}{2i|x|^2})e^{i|x|^2/t}
-\int_0^\infty t^2\frac{d}{ds}(\frac{1}{2i(s+|x|)}\frac{d}{ds}(\frac{e^{-s}}{2i(s+|x|)}))e^{i(s+|x|)^2/t}ds. $$
Write $I(t)=I_1(t)+I_2(t)+I_3(t)$, corresponding to the three terms above.
We need oscillation in $x$ to bound $I_1$; we don't need it for $I_2$ and $I_3$.
$$I_1(t)=\frac{t}{2i}\int \frac{e^{i|x|^2/t}}{|x|}f(x)dx=-\frac{t}{2i}\int e^{i|x|^2/t}\vec x\cdot\nabla(\frac{f(x)}{|x|(3+2i|x|^2/t)})$$
is bounded by a constant (depending on $t$) times
$$\int \frac{|\nabla f(x)|}{|x|^2}+\frac{|f(x)|}{|x|^3}\le(\int |\nabla f|^2)^{1/2}(\int_{|x|\ge1} |x|^{-4})^{1/2}+(\int |f|^2)^{1/2}(\int_{|x|\ge1} |x|^{-6})^{1/2}\le C.$$
For $I_2$ we have
$$ |I_2(t)|\le t^2\int |x|^{-2}f(x)\le t^2(\int |f|^2)^{1/2}(\int_{|x|\ge1} |x|^{-4})^{1/2}.$$
For $I_3$ we note that both $e^{-s}$ and $1/(s+|x|)$ are decreasing and convex in $s$, so is their product. Hence $\frac{d}{ds}\frac{e^{-s}}{s+|x|}\le0$ and is increasing. Since $1/(s+|x|)>0$ is decreasing, their product is negative increasing. Hence
$$ \int_0^\infty |\frac{d}{ds}(\frac{1}{2i(s+|x|)}\frac{d}{ds}(\frac{e^{-s}}{2i(s+|x|)}))|ds=\frac{t}{2|x|}(\frac{t}{2|x|}+\frac{t}{2|x|^2}) $$
and the bound for $I_3$ follows the same way as $I_2$.
P.S. Going through the above estimates with an explicit dependence on $t$ shows that the bound growth like $t^2$, though I think the actually growth may well be slower.
P.P.S. A place to study oscillatory integrals systematically is Chapter 6 of Stein, Harmonic Analysis.