The answer to the question in the first sentence is "yes". Let $M$ be a hyperbolic 3-manifold whose isometry group is trivial. Then by Theorem 1.1 of

Farb, Benson; Weinberger, Shmuel Hidden symmetries and arithmetic manifolds. Geometry, spectral theory, groups, and dynamics, 111–119, Contemp. Math., 387, Amer. Math. Soc., Providence, RI, 2005. (Reviewer: Bachir Bekka) 53C35 (53C23)

(which they attribute to Borel, though they do not give an original reference for it), the isometry group of every Riemannian metric on $M$ is isomorphic to a subgroup of the hyperbolic isometry group, and thus is trivial.

Along similar lines, you might be interested in Theorem H of

Dinkelbach, Jonathan and Leeb, Bernhard,
Equivariant Ricci flow with surgery and applications to finite group actions on geometric 3-manifolds.
Geom. Topol. 13 (2009), no. 2, 1129–1173.

It says that every finite-order diffeomorphism of a closed hyperbolic 3-manifold is smoothly conjugate to an isometry, so closed hyperbolic 3-manifolds with trivial isometry groups give examples of smooth manifolds with torsion-free diffeomorphism groups.