Let $f(x): \mathbb{R} \to \mathbb{R}_{\geq 0}$ be a smooth function. I am interested in estimating sums of the form $$ \sum_{ A < n \leq B } \{ f(n)\} $$ where $\{ c \}$ denotes the fractional part of $c \in \mathbb{R}$. Trivial upper bound is $B-A$. Are there any known non-trivial upper bounds for 'nice' $f$? say for example something like a polynomial or $f(x) = \sqrt{x}$ or $B/x$?

You should look at Equidistribution Theory, in particular equidistribution modulo 1. Equidistribution can be checked by confirming that $$\lim_{n\rightarrow \infty}\sum_{i=1}^n e^{2 \pi i f(n)}=0.$$

Then you should have $$\sum_{A<n<B}\{f(n)\}=(B-A)/2+o(B-A)$$ which gives you an upper bound of $$|\sum_{A<n<B}\{f(n)\}|\leq|B-A|(1/2+\epsilon)$$ for all $\epsilon>0$ and large enough $B-A$.

In particular this will hold if $f$ is a polynomial with at least one coefficient other than the constant term irrational. Also $\{x^n\}$ is equidistributed for almost all $x\in \mathbb{R_{>1}}$

A sum of fractional parts (1967) contains some results that may be of interest.

If $A=0$ and $B$ is a prime $p$, and $f(x)$ is a polynomial with integer coefficients, which does not vanish identically modulo $p$, then
$$\sum_{n=0}^{p-1}\{f(n)/p\}=\tfrac{1}{2}p+{\cal O}(\sqrt{p}\log p).$$