Yes, such exponential sums can be estimated using the continued fraction coefficients of $\alpha$. There is a field of mathematics called uniform distribution theory which is concerned with the distribution of sequences in the unit interval. The degree of uniformity is measured by something called discrepancy. The infinite sequence of fractional parts of $n \alpha$ is called Kronecker sequence. There are explicit estimates for the discrepancy of Kronecker sequences in terms of continued fraction coefficients.

These discrepancy estimates can be turned into bounds for exponential sums (or sums of other 1-periodic functions) by the Koksma-Hlawka inequality. See for example here: https://en.wikipedia.org/wiki/Low-discrepancy_sequence. The standard reference is the book of Kuipers and Niederreiter, *Uniform Distribution of Sequences*, see page 122 ff. for Kronecker sequences.

To answer your particular question, if $\alpha$ is badly approximable then the continued fractions coefficients are bounded, which implies that the discrepancy is of order $(\log N)/N$, which implies that the exponential sum is of order $\log N$.