Let $N$ be a compact smooth manifold. By "mapping class group" I will mean
$$\pi_0 \operatorname{Diff}(N)$$
i.e. the isotopy-classes of diffeomorphisms of $N$.
My presumption is that this mapping class group is likely to be finitely generated if and only if either $n = \dim N \leq 3$ or when $n \geq 4$ the group $\pi_1 N$ has finitely-many conjugacy classes, i.e. the path-components of the free loop space on $N$ are finite.
i.e. other than in low dimensions, finite-generation of the mapping class group corresponds to the fundamental group being finite.
Are there any obvious counter-examples to this statement I might be missing? I'm wondering if it's a reasonable conjecture or not. I would imagine this conjecture would remain the same in the PL and topological categories. At least, all the non-finite-generation examples I know of are category-independent, and smoothing theory I believe would say this question is category-independent in high dimensions.