$\newcommand{\la}{\lambda}\newcommand{\Ga}{\Gamma}$Write
\begin{equation*}
I(\la)=\int_0^\infty dx\,f(x)J(\la x),
\end{equation*}
where
\begin{equation*}
J(z):=\int_0^1 dt\,e^{i z t^a}=\frac b{z^b}K(z),
\end{equation*}
\begin{equation*}
K(z):=\int_0^z du\,u^{b-1}e^{i u},
\end{equation*}
$b:=1/a\in(0,1)$.
Note that
\begin{equation*}
K(z)\to\int_0^\infty du\,u^{b-1}e^{i u}=(-i)^{-b}\Ga(b)
\end{equation*}\begin{equation*}
K(z)\to\int_0^\infty du\,u^{b-1}e^{i u}=(-i)^{-b}\Ga(b) \tag{0}\label{0}
\end{equation*}
as $z\to\infty$. (The equality in \eqref{0} can be obtained in a number of ways; in particular, it follows immediately from formulas 3.761.4 and 3.761.9 of Gradshteyn and Ryzhik, 7th Edition.)
Also, since $u^{b-1}$ decreases to $0$ as $u$ increases from $0$ to $\infty$, we have $|K(z)|\le C$ for some real $C>0$ and all real $z\ge0$.
So, letting $\la\to\infty$, by dominated convergence we get
\begin{equation*}
\la^b I(\la)=\int_0^\infty dx\,f(x)\frac b{x^b}K(\la x) \\
\to (-i)^{-b}\Ga(b+1)\int_0^\infty dx\,f(x)/x^b,
\end{equation*}
so that
\begin{equation*}
I(\la)\sim R(\la):=(-i\la)^{-b}\,\Ga(b+1)\int_0^\infty dx\,f(x)/x^b. \tag{1}\label{1}
\end{equation*}
Here are the graphs $\{(\la,\Re\frac{I(\la)}{R(\la)})\colon0<\la\le20\}$ (black) and $\{(\la,\Im\frac{I(\la)}{R(\la)})\colon0<\la\le20\}$ (blue) for $a=2.1$ and $f(x)=\exp(-\frac1{x(1-x)})\,1(0<x<1)$, which confirm \eqref{1}:
