Ok, apologies if this is overkill, but this paper shows that for almost every (irrational) $\alpha \in \mathbb{R}$, only a measure $0$ set of $x \in \mathbb{R}$ satisfy $S_{N,\alpha}(x) = o(\sqrt{N})$ as $N \to \infty$.
Indeed, that paper,
Metric theory of Weyl sums
Changhao Chen, Bryce Kerr, James Maynard, Igor Shparlinski
shows that there is an absolute constant $c > 0$ such that almost every pair $(x,\alpha) \in \mathbb{R}^2$ has $|S_{N,\alpha}(x)| \ge c\sqrt{N}$ for infinitely many $N$. It follows from basic measure theory that for almost every $\alpha \in \mathbb{R}$, it holds for almost every $x \in \mathbb{R}$ that $|S_{N,\alpha}(x)| \ge c\sqrt{N}$ for infinitely many $N$.