ThisThe answer to the question in the first sentence is equivalent to finding smooth manifolds"yes". Let $M$ be a hyperbolic 3-manifold whose diffeomorphismisometry group is trivial. Then by Theorem 1.1 of
Farb, Benson; Weinberger, Shmuel Hidden symmetries and arithmetic manifolds. Geometry, spectral theory, groups are torsion-free; indeed, by averaging one can see that every finite-order diffeomorphismand dynamics, 111–119, Contemp. Math., 387, Amer. Math. Soc., Providence, RI, 2005. (Reviewer: Bachir Bekka) 53C35 (53C23)
(which they attribute to Borel, though they do not give an original reference for it), the isometry group of a smooth manifold preserves someevery Riemannian metric on $M$ is isomorphic to a subgroup of the hyperbolic isometry group, and thus is trivial.
Along similar lines, you might be interested in Theorem H of
Dinkelbach, Jonathan and Leeb, Bernhard, Equivariant Ricci flow with surgery and applications to finite group actions on geometric 3-manifolds. Geom. Topol. 13 (2009), no. 2, 1129–1173.
It says that every finite-order diffeomorphism of a closed hyperbolic 3-manifold is smoothly conjugate to an isometry, so closed hyperbolic 3-manifolds with trivial isometry groups give examples of smooth manifolds with torsion-free diffeomorphism groups.