I believe now that David Speyer's example can be adapted to provide a counterexample to the original question. (So I retract my earlier comment on the question and will delete it soon.)

In David's example, $A$ is a degree $\phi(m)$ matrix of order $m$ defining the action by multiplication of $\zeta_m$ on the ideal $I$ of the number field ${\mathbb Q}[\zeta_m]$, and $B$ is the action on the ideal $\sigma(I)$, and $A$ and $B$ are not conjugate to each other or to their inverses in ${\rm GL}_{\phi(m)}({\mathbb Z})$. A specific example is $m=37$, $\phi(m)=36$.

We define degree $n:=\phi(m)+1$ matrices $A'$ and $B'$
as the diagonal joins of $A$ and $B$ with the identity matrix $I_1$. So the corresponding ${\mathbb Z}$-modules can be thought of as $I \oplus \langle y \rangle$ and $\sigma(I) \oplus \langle z \rangle$, with trivial action on the second factors. These modules cannot be isomorphic, because an isomorphism would have to map the fixed points submodule $\langle y \rangle$ onto $\langle z \rangle$ and then their quotients $I$ and $\sigma(I)$ would be isomorphic, which they are not. So $A'$ and $B'$ are not conjugate in ${\rm GL}_{n}({\mathbb Z})$.

I claim (at least in some cases) that we can choose $A$ and $B$ such that the corresponding semidirect products $\langle \alpha \rangle \ltimes_{A'} {\mathbb Z}^n$ and $\langle \beta \rangle \ltimes_{B'} {\mathbb Z}^n$ are isomorphic, where $\alpha$ and $\beta$ generate infinite cyclic groups. We can (in some cases?) choose $A = B^a$ with $a$ coprime to $m$ and $2 \le a < \phi(m)-1$ such that $B$ is not conjugate in ${\rm GL}_{\phi(m)}({\mathbb Z})$ to $A$ or to $A^{-1}$, and choose integers $r,s$ with $ra-sm=1$.

Then we can define a isomorphism from $\langle \alpha \rangle \ltimes_{A'} {\mathbb Z}^n$ to $\langle \beta \rangle \ltimes_{B'} {\mathbb Z}^n$ by mapping $I$ to $\sigma(I)$ as in David's example, $y$ to $\beta^m z^r$ and $\alpha$ to $\beta^a z^s$. Note that this induces an isomorphism from the free abelian group $\langle \alpha, y \rangle$ to $\langle \beta, z \rangle$, such that the image of $y$ centralizes $\sigma(I)$.

I did some calculations in Magma in the case $m=37$, and found a degree 36 integer matrix $A$ that is not conjugate to $A^a$ for any $a$ with $2 \le a \le 36$.

For completeness, here is the matrix $A$ in machine readable format. I used the Magma function $\mathsf{AreGLConjugate}$ to test $A$ for conjugacy with $A^i$. This uses a fairly new algorithm published in Bettina Eick, Tommy Hofmann, and E.A. O'Brien.
The conjugacy problem in ${\rm GL}(n,{\mathbb Z})$.
J. London Math. Soc., 2019.

```
[
[-4,149,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-1,4,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-4,133,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-2,64,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-2,42,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-4,130,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-3,76,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-4,143,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-1,24,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-2,53,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-3,86,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-3,103,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-1,35,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-1,9,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-4,113,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-4,144,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-1,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-2,69,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-1,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-2,61,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-2,54,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-3,82,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0],
[-4,119,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0],
[-4,120,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0],
[-4,116,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0],
[-4,132,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0],
[-2,68,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0],
[-1,26,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0],
[-2,45,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0],
[-4,118,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0],
[-4,124,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0],
[-3,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0],
[-2,47,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0],
[-3,110,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0],
[-1,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],
[15,120,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
]
```