There is a little room for repeat eigenvalues, as long as we have nontrivial Jordan blocks. For the following, if an integral square matrix commutes with $A_j,$ it is a (rational) polynomial in $A_j$: $$ A_2 \; = \; \left( \begin{array}{rr} 1 & 1 \\\ 0 & 1 \end{array} \right) , $$
$$ A_3 \; = \; \left( \begin{array}{rrr} 1 & 1 & 0 \\\ 0 & 1 & 1 \\\ 0 & 0 & 1 \end{array} \right) , $$
$$ A_4 \; = \; \left( \begin{array}{cccc} 0 & -1 & 1 & 0 \\\ 1 & 0 & 0 & 1 \\\ 0 & 0 & 0 & -1 \\\ 0 & 0 & 1 & 0 \end{array} \right). $$
EDIT : it seems reasonable to conjecture that the full set of $A \in SL_n(\mathbb Z)$ for which the statement is true is $ A \in SL_n(\mathbb Z)$ such that, should there be any eigenvalue(s) of multiplicity larger than one, all occurrences of that eigenvalue must fit into a single Jordan block. Richard would know.