Let $\pi_1$ be the fundamental group of a closed aspherical manifold of dimension $n$. In particular, $\pi_1$ is finitely presented, torsion-free and its cohomology is finitely generated and satisfies Poincaré duality.

Now, we take the center $\mathcal{Z}\pi_1$ (which I assume is finitely generated) and consider the inner automorphism group $Inn(\pi_1)=\pi_1/\mathcal{Z}\pi_1$. I would like prove that $Inn(\pi_1)$ always have an element of infinite order unless $\pi_1\cong\mathbb{Z}^n$.

By a theorem of Schur, if $Inn(\pi_1)$ is finite then $[\pi_1,\pi_1]$ is also finite and therefore $[\pi_1,\pi_1]$ is trivial, since $\pi_1$ is torsion-free. In this case $\pi_1$ is abelian and hence $\pi_1\cong \mathbb{Z}^n$ and the manifold is a torus.

So we only need to exclude the case where $Inn(\pi_1)$ is infinite periodic group (all elements are torsion elements).

Note that if $\pi_1$ is centerless then $Inn(\pi_1)\cong\pi_1$ which is torsion-free, so I was looking onto cases where the center is large (for example $\mathbb{Z}^{n-1}$).

I was mainly trying to use some cohomological arguments. For example, $H_2(Inn(\pi_1),\mathbb{Z})$ needs to be finitely generated. In this paper, they prove that $H_2(B(a,b),\mathbb{Z})$ of the free Burnside group with odd $b\geq 665$ has countable rank (in conseqeunce $Inn(\pi_1)\ncong B(a,b)$). A similar argument from this paper, can be used to show that $Inn(\pi_1)$ cannot be an infinite 2-group of bounded exponent.

Finally, $Inn(\pi_1)$ is finitely presented, but it seems that the Burnside problem for finitely presented groups is still open (per this mathoverflow question).

Maybe there is a trivial reason for $Inn(\pi_1)$ to have always elements of infintie order that has slipped out of my mind, but I don't see it rigth now.