The precise question is the following:
Is there a closed aspherical manifold $M$ of dimension $n\geq 3$ such that Out($\pi_1(M)$) is infinite and $\pi_1(M)$ does not contain $\mathbb Z \times \mathbb Z$?
Every surface of genus $g\geq 2$ is of course an example of such manifold, but I cannot find any other example in higher dimension. Note that $\pi_1(M)$ cannot be hyperbolic, otherwise Out($\pi_1(M)$) would be finite. There are no examples in dimension $n=3$ because of Geometrization. I cannot find any reference to this natural question in the literature, but I am probably missing something.