Claim. The groups $G_A$ and $G_B$ are not isomorphic.
We will use the following lemmas.
Lemma 1. Let $A \in \text{GL}_n(\mathbb{Z})$ and let $G_A \Doteq \mathbb{Z} \ltimes_A \mathbb{Z}^n$. Then the following hold:
- The center $Z(G_A)$ of $G_A$ is generated by $\{0\} \times \ker(A - 1_n)$ and $(\omega, (0, \dots, 0))$ where $1_n$ is the $n \times n$ identity matrix and $\omega$ is the order of $A$ in $\text{GL}_n(\mathbb{Z})$ if $A$ has finite order, zero otherwise.
- The derived subgroup $[G_A, G_A]$ of $G_A$ is $\{0\} \times (A - 1_n)\mathbb{Z}^n$. More generally, setting $\gamma_{i + 1}(G_A) \Doteq [\gamma_i(G_A), G_A]$ with $\gamma_1(G_A) \Doteq G_A$, we have $\gamma_{i + 1}(G_A) = \{0\} \times (A - 1_n)^i \mathbb{Z}^n$.
Proof. Straightforward.
For $A$ and $B$ as in OP's question, we have thus $$Z(G_A) = 4\mathbb{Z} \times \ker(A - 1_5), \, Z(G_B) = 4\mathbb{Z} \times \ker(B - 1_5)$$
with $\ker(A - 1_5) = \ker(B - 1_5) = \mathbb{Z} \times \{ (0, 0, 0, 0) \} \subset \mathbb{Z}^5$.
Lemma 2. Let $A$ and $B$ as in OP's question and set $\Gamma_A \Doteq G_A / Z(G_A)$ and $\Gamma_B \Doteq G_B / Z(G_B)$. Then we have $\Gamma_A/ [\Gamma_A, \Gamma_A] \simeq (\mathbb{Z}/ 2 \mathbb{Z})^3 \times \mathbb{Z}/ 4 \mathbb{Z}$ and $\Gamma_B/ [\Gamma_B, \Gamma_B] \simeq \mathbb{Z}/ 2 \mathbb{Z} \times (\mathbb{Z}/ 4 \mathbb{Z})^2$.
Proof. Write $\Gamma_A = \mathbb{Z} / 4 \mathbb{Z} \ltimes_{A'} \mathbb{Z}^4$ and
$\Gamma_B = \mathbb{Z} / 4 \mathbb{Z} \ltimes_{B'} \mathbb{Z}^4$ where $A', B' \in \text{GL}_4(\mathbb{Z})$ are obtained from $A$ and $B$ by removing the first row and the first column. Use then the description of the derived subgroup of Lemma 1 which still applies to $\Gamma_A$ and $\Gamma_B$ if we replace $A$ by $A'$ and $B$ by $B'$.
Proof of the claim. If $G_A$ and $G_B$ are isomorphic, then so are $\Gamma_A$ and $\Gamma_B$. This is impossible since the two latter groups have non-isomorphic abelianizations by Lemma 2.
Addendum. Let $C_A$ be the cyclic subgroup of $G_A$ generated by $a \Doteq (1, (0, \dots, 0))$ and $K_A$ the $\mathbb{Z}[C_A]$-module defined as in Johannes Hahn's answer (and subsequently mine) to this MO question.
Let $\omega(A)$ be the order of $A$ in $\text{GL}_n(\mathbb{Z})$, that we assume to be finite, and set $e_0 \Doteq (\omega(A), (0, \dots, 0)) \in G_A$. Let us denote by $(e_1, \dots, e_n)$ the canonical basis of $\mathbb{Z}^n \triangleleft G_A$.
It has been established that the pair $\{K_A, K_{A^{-1}}\}$ of $\mathbb{Z}[C]$-modules is an isomorphism invariant of $G_A$, where $C = C_A \simeq C_{A^{-1}}$ with the identification $a \mapsto (1, (0, \dots,0)) \in G_{A^{-1}}$.
For the instances of this MO question, straightforward computations show that
$$\left\langle e_0, e_2, e_3, e_5 \, \vert \, (a - 1)e_0 = (a + 1)e_2 = (a + 1)e_3 = (a^3 -a^2 + a - 1)e_5 = 0\right\rangle$$ is a presentation of both $K_A$ and $K_{A^{-1}}$ and
$$\left\langle e_0, e_1, e_2, e_3, e_5 \, \vert \,
(a - 1)e_0 = (a -1)e_1 = (a + 1)e_2 = (a + 1)e_3 = (a^2 + 1)e_5 + e_1 + e_2 = 0\right\rangle$$ is a presentation of $K_B$.
From the above presentations, we easily infer the following isomorphisms of Abelian groups: $K_A/(a + 1)K_A \simeq \mathbb{Z}/2\mathbb{Z} \times \mathbb{Z}/4 \mathbb{Z} \times \mathbb{Z}^2$ and $K_B/(a + 1)K_B \simeq (\mathbb{Z}/2\mathbb{Z})^2 \times \mathbb{Z}^2$.
As result, the groups $G_A$ and $G_B$ are not isomorphic.