I found a proof here for $n=4$:
Yang, Qingjie, Conjugacy classes of torsion in (\mathrm{GL}_N(\mathbb Z)), Electron. J. Linear Algebra 30, 478-493 (2015). ZBL1329.15063. MR3414308
See the discussion in the last paragraph on p. 482 for the case that the characteristic polynomial is irreducible, and Theorem 1.7 for the reducible case.
On the other hand, I suppose it's possible that the number of conjugacy classes of finite-order elements in $GL_n(\mathbb{Z})$ could grow faster than the number of $0,\pm1$ matrices intersected with $GL_n(\mathbb{Z})$. One can get a lower bound on the number of conjugacy classes of finite-order elements in $GL_n(\mathbb{Z})$ by counting the number which are block-diagonalizable with irreducible blocks. This should correspond to a sum over decompositions of $n$ into $\varphi(m)$ by $|Cl(\mathbb{Z}[e^{2\pi i/m}])|$, a sum over class numbers, since one obtains a conjugacy class of element of $GL_{\varphi(m)}(\mathbb{Z})$ of order $m$ for every ideal class in $\mathbb{Z}[e^{2\pi i/m}]$. I have no intuition though for the growth of this function, especially since the class numbers of cyclotomic fields behave erratically.