I think that the construction in Belolipetsky--Lubotzky, Finite Groups and Hyperbolic Manifolds (https://arxiv.org/abs/math/0406607) provides a more algebraic construction of such manifolds for any $n$.
One of the results in this paper (Theorem 3.1) is the following : given groups $\Gamma \triangleright \Delta \triangleright M$ with $\Gamma/\Delta$ finite and $F = \Delta/M$ a nonabelian free group, there exists $\Gamma \ge D \ge \Delta$ and $B \le D$ of finite index with $N_\Gamma(B) = B$ (in fact they prove that for any finite group $G$ there are infinitely many $B$ with $N_\Gamma(B)/B$ isomorphic to $G$).
They use this to construct closed hyperbolic manifolds with trivial isometry group ; by Mostow rigidity the mapping class group of $X$ is isomorphic to the isometry group of $X$ (every homeomorphism is isotopic to an isometry), so these manifolds will have trivial mapping class group. Without too much details their construction uses a specific lattice $\Gamma$ in the isometry group of hyperbolic $n$-space $\mathbb H^n$, and subgroups $D, \Delta, M, B$ as in the previous paragraph ; the manifold is the quotient of $\mathbb H^n$ by $B$ (their construction ensures that $B$ is torsion-free though $\Gamma$ will likely not be).
Now to look at Betti numbers, for which we need to dive a bit more into the details of this paper. Since $X$ is aspherical we have
$$b_1(X, \mathbb Q) = b_1(B, \mathbb Q) \ge b_1(D, \mathbb Q).$$
On the other hand $b_1(D, \mathbb Q)$ is the dimension of the fixed subspace of $D/\Delta$ in $H^1(\Delta, \mathbb Q)$ (conjugation action on the normal subgroup $\Delta$ and its morphisms to $\mathbb Z$). By the definition of $D$ in the paper of Belolipetsky--Lubotzky (p. 4 in the arxiv version) we have that it acts trivially on the part of the cohomology coming from the surjective morphism $\Delta \to F$. So we have that $b_1(X, \mathbb Q)$
is larger than the rank of $F$. It remains to remark that in the data for Theorem 3.1 the quotient $\Delta/M$ can be taken to have arbitrarily large rank (since a non-abelian free group contains free subgroups of finite index with arbitrarily large rank), so we get examples with $b_1(B, \mathbb Q)$ arbitrarily large.