Let $M$ be a manifold with boundary. Consider the following groups:

(1) $\pi_0(\operatorname{Diff}(M,\partial M))$.

(2) $\pi_0(\operatorname{Homeo}(M,\partial M))$.

(3) $\pi_0(\operatorname{HomEq}(M,\partial M))$.

That is, isotopy (resp. isotopy and homotopy) classes of diffeomorphisms (resp. homeomorphisms and homotopy equivalences) relative to $\partial M$.

I would like to find conditions on $M$ which guarantee that one of the groups (1)/(2)/(3) has no element of finite order.

My motivation for this is to generalize from the case $\dim M=2$, when as long as $\partial M\ne\varnothing$, none of the groups (1)/(2)/(3) has an element of finite order (This is left as an exercise for the reader. Hint: an element of finite order in $\operatorname{MCG}(\Sigma_g)$ fixes some hyperbolic structure).

Of course, if $\dim M=2$, then (1)=(2)=(3). This question is really about finding a proper generalization of the result for $\dim M=2$ to higher dimensions, so I'm leaving it open as to which of (1)/(2)/(3) this question is really about. We remark that (1) seems unlikely to be the right group to consider; for instance when $(M,\partial M)=(D^n,S^{n-1})$ and $n>4$, then it is the group of exotic spheres in dimension $n+1$.