Here is an approach which may give some better estimates for particular values of $\alpha$:

$$\sum_{i= 1}^N i((i\alpha)) = \sum_{i=1}^N \sum_{j=i}^N ((j \alpha)) = \sum_{i=1}^N \sum_{k=0}^{N-i}((N\alpha -k\alpha))$$.

So, if you can estimate

$$\sup_{\substack{x\in (0,1) \\\ M \le N}} \bigg|\sum_{k=0}^{M} ((x-k\alpha))\bigg|,$$

then you can crudely multiply by $N$ to get an estimate for $|\sum i((i\alpha))|$.

Specifically, my guess is that for quadratic irrationals $\alpha$, there is an upper bound for

$$\bigg|\sum_{k=0}^M ((x-k\alpha))\bigg|$$

which is $O (\log M)$, which would give you a bound of $O(N \log N)$, and more generally that there is a bound in terms of the coefficients of the simple continued fraction for $\alpha$, so that if those are bounded, then you still get $O(N \log N)$.

For the particular value $\phi = (\sqrt5 + 1)/2$, $\sum_{i=0}^{M} ((i \phi))$ has logarithmic growth $c + (5\sqrt5 - 11)/4 \log_\phi M$ (achieved at indices in the sequences A064831 (+) and A059840 (-)), which suggests that $\sup \sum_{i=0}^M ((x-i\phi))$ also has logarithmic growth, which would give an $N \log N$ bound for the sum.

In the opposite direction, for all $\alpha \not\in \frac 12\mathbb Z$, $$\limsup \bigg(\log_N \bigg|\sum_{i=0}^N i((i\alpha))\bigg|\bigg) \ge 1$$ since there are terms proportional to $N$.

The sum can be greater than $N^{2-\epsilon}$ infinitely often by choosing $\alpha$ so that it is extremely well approximated by infinitely many rational numbers. When $\alpha$ is very closely approximated by $p/q$, then for $N$ a small multiple of $q$ (where "small" is relative to how well $p/q$ approximates $\alpha$), about $1/q$ of the terms can be moved past integers with a small perturbation of $\alpha$ to $\alpha'$, which causes a jump of about $N^2/q$ in the sum. So, either the sum for $\alpha$ or $\alpha'$ is large. We can choose a sequence $p_n/q_n$ which converges to an $\alpha$ which produces large sums infinitely often, so that for these $\alpha,$
$$\limsup \bigg(\log_N \bigg|\sum_{i=0}^N i((i\alpha))\bigg|\bigg) = 2$$.