There are situations in which surfaces are the "unique" examples with big mapping class groups. One such is closed manifolds of negative sectional curvature.
Theorem (Paulin): If $M$ is a closed $n$-dimensional manifold of negative sectional curvature and $n>2$ then $\mathrm{Out}(\pi_1M)=\mathrm{MCG}(M)$ is finite.
Proof: Paulin actually proved that if $\Gamma$ is a torsion-free word-hyperbolic group (such as the fundamental group of a closed manifold of negative curvature) and $\mathrm{Out}(\Gamma)$ is infinite then $\Gamma$ splits over a cyclic subgroup, say as $\Gamma=A*_CB$ for $C$ cyclic. (There is also the HNN-extension case, which is similar.)
Mayer--Vietoris now gives that
$\ldots\to H^{n-1}(C)\to H^n(M)\to H^n(A)\oplus H^n(B) \to H^n(C)\to\ldots$
is exact.
Since $n>2$, $H^{n-1}(C)\cong H^n(C)\cong 0$ and so $H^n(M)\cong H^n(A)\oplus H^n(B)$. But Strebel's theorem implies that $H^n(A)\cong H^n(B)\cong 0$ , a contradiction. QED
(Here, I'm taking $\mathrm{MCG}(M)$ to mean $\pi_0$ of self-homotopy-equivalences. Since negatively curved manifolds are aspherical, this coincides with $\mathrm{Out}(\pi_1M)$.)
This is actually an instance of a much more general phenomenon. In good situations, JSJ theory tells us that $\mathrm{Out}(\Gamma)$ can be 'broken up' into pieces which are essentially mapping class groups of surfaces and tori. This happens in the case of irreducible 3-manifolds, for instance.