I can't help thinking that the answer ought to come out of a combination of standard techniques. First, indeed, reduce to fractional parts: but with a caveat, that it would be sensible to do some manipulation first. I'm thinking about "summation by parts".
Edit: There is a classic reference: G. H. Hardy & J. E. Littlewood, "Some problems of Diophantine approximation: The lattice points of a right-angled triangle," (1st memoir), Proc. London Math. Soc. (2), v. 20, 1922, pp. 15-36. They consider the modified fractional part sum, with {x} set as x - [x] - 1/2, of the ${k\theta}$ up to n, where to be compatible with their notation $\theta$ would be the reciprocal of $\tau$, not that this matters at all. The bound they get is O(log n) (Hardy's Works vol. I p. 145), which depends only on the continued fraction having bounded partial quotients. The particular case relevant to $\tau$ is worked out in detail over the next few pages. The result is sharp. Off the top of my head this looks enough to get the error term O(nlog n) for the sum as posted, by breaking into at most n sums of this type.