There is a little room for repeat eigenvalues, as long as we have nontrivial Jordan blocks. For the following, if an integral square matrix commutes with $A_j,$ it is a (rational) polynomial in $A_j$: 
$$ A_2 \; = \;  
 \left(  \begin{array}{rr}
  1 & 1  \\\
   0  & 1  
\end{array} 
  \right)  ,
  $$


$$ A_3 \; = \;  
 \left(  \begin{array}{rrr}
  1 & 1 & 0 \\\
  0  & 1 & 1 \\\
  0 & 0  & 1  
\end{array} 
  \right)  ,
  $$

$$ A_4 \; = \;  
 \left(  \begin{array}{cccc}
 0 & -1 & 1 & 0 \\\
 1 & 0  & 0 & 1 \\\
 0 & 0 & 0  & -1  \\\ 
 0 & 0 & 1  & 0  
\end{array} 
  \right).
  $$

 EDIT : it seems reasonable to conjecture that the full set of $A \in SL_n(\mathbb Z)$ for which the statement is true is $ A \in SL_n(\mathbb Z)$ such that, should there be any eigenvalue(s) of multiplicity larger than one, all occurrences of that eigenvalue must fit into a single Jordan block.