My question is if it is possible to find a compactly supported smooth function $\varphi:\mathbf{R}\to \mathbf{R}$ s.t. the following integration $\int_{\mathbf{R}}\varphi(t)e^{itx}e^{tx}dt$ stays bounded for $x\in \mathbf{R}$.
No. This follows from the PhragmenLindelof Theorem.
EDIT. Consider the function of a complex variable $z$, $$F(z)=\int_R\phi(t)e^{itz}dt.$$ This function is bounded on the real axis by $\\phi\_1$. It is also of exponential type, $\logF(z)\leq O(z),\; z\to\infty$. Your condition says that $F(z)$ is bounded on the line $\{ z=xix:x\in R\}$. Then PhragmenLindelof says that $F$ must be bounded, thus constant. But this is impossible if $\phi$ is smooth.

$\begingroup$ In en.wikipedia.org/wiki/Phragmén–Lindelöf_principle, the PhragménLindelöf Principle is about analytic functions. Maybe add an explanation showing how it applies here. $\endgroup$ – Gerald Edgar Dec 18 '13 at 19:43

$\begingroup$ Sorry I don't quite get it, could you say a bit more? How do we get the contradiction from the Phragmen–Lindelof principle? It just says the function is bounded, right? $\endgroup$ – Shaoming Dec 18 '13 at 19:46

$\begingroup$ The Fourier transform of $\phi$ is an entire function with exponential growth which is bounded on the real axis. If it is also bounded on another line through the origin, then the PhragmenLindelof Principle forces it to be constant. This allows only the trivial case $\phi=0$. $\endgroup$ – Michael Renardy Dec 18 '13 at 22:39

$\begingroup$ I don't see how $F$ being bounded implies $F$ being constant. For example $e^{z^2}$ is bounded on the cone with forms an angle less than 45 degrees with the horizontal axis, but is not constant. $\endgroup$ – Shaoming Dec 19 '13 at 14:27