The answer to your question is yes, and it is a pretty well-understood topic.
First of all $X$ is more or less irrelevant for the bounds in 4) so let us take $X = Y$, say, for convenience.
Second, we can always scale everything down by $Y$. So without loss of generality $Y = 1$.
Put $g = f'$. Let us also for simplicity assume that $f$ is symmetric so it is enough to study $g$ on $[1, 2]$. The problem therefore is reduced to the following: for which sequences $t_j$ we can find a function $g$ such that $g = 0, x\notin [1, 2]$, $\int g = 1$ and $|g^{j}(x)|\le C t_j$ for some constant $C$.
The answer to this question in more or less full generality is given by the Denjoy-Carleman Theorem: if the sequence $M_j = \frac{t_j}{j!}$ is logarithmically convex (i.e. $\frac{M_{j+1}}{M_j}$ is increasing in $j$) then such a function exists if and only if $\sum_j \frac{1}{jM_j^{1/j}} < \infty$. For example there exists a function $f$ such that
\begin{equation}\label{bound}
|f^{(j)}(x)| \le CY^{-j}j^{(1+\varepsilon)j}
\end{equation}
for any fixed $\varepsilon > 0$ (this is related to the so-called Gevrey classes).
Actually, since you mentioned Fourier transform, let me write about a different result which is more directly applicable to this type of problems: Beurling-Malliavin multiplier theorem. It reads as follows:
Let $w:\mathbb{R}\to \mathbb{R}$ be a nonnegative Lipshitz function (this is a small technical condition). Then there exists a nonzero compactly supported function $f$ with $|\hat{f}(\xi)| \le e^{-w(x)}$ if and only if integral
$$\int_\mathbb{R} \frac{w(x)}{x^2 + 1}dx$$
is convergent. Moreover, support of the function can be arbitrary small.
Lastly, if you want an explicit function $g$ (and therefore $f$), satisfying the above bound, you can take
$$g(x) = e^{-(1-x)^{-m}}e^{-(x+1)^{-m}}\chi_{(-1, 1)},$$
see e.g. this paper, section 3.1.