Here's a proof that a finite $p$-group $G$ that has property $A(G)$ is abelian, as claimed in the comments. Below, I have added a slightly sketchy proof that any $G$ with property $A(G)$ is either abelian, or the product of an abelian group of order prime to 6 and one of the groups $G_k$ mentioned in the question.
The $p$-group case: Let $G$ be a $p$-group of minimal order such that $A(G)$ holds and $G$ is not abelian. Then (as pointed out in the comments) every quotient group of $G$ has property $A$ so is abelian. It follows that $G^\prime$ is the unique minimal normal subgroup of $G.$ Since in particular $G^\prime\subseteq Z(G),$ for any $a,b\in G$ and $n\in\mathbb Z$ we have $$(ba)^n=b^na^n[a,b]^{\tfrac 12n(n-1)}.$$ As $A(G)$ holds, there is $n$ such that $$(ba)^n=[a,b]^{{\tfrac 12n(n-1)}+1}.$$
First consider this when $p>2.$ Whenever $a$ and $b$ do not commute, this equation cannot have a common value $1,$ since that would require $n$ prime to $p,$ and the contradiction $ba=1.$ Since $|G^\prime|=p,$ we have $G^\prime\subseteq \langle ba\rangle.$ Now if $g\in G$ is any element, either $g\in Z(G)$ or $g=ba$ for some $a$ and $b$ that do not commute. We conclude that $\langle g\rangle$ is normal in $G$ for all $g\in G.$ Thus every subgroup of $G$ is normal ($G$ is a Dedekind group). Such a group is either abelian or the direct product of $Q_8$ and an abelian group. As $|G|$ is odd, $G$ is abelian, a contradiction.
Now assume $p=2.$ Suppose $g\in G-Z(G)$ has order 4. Again we can write $g=ba$ where $a$ and $b$ do not commute. If $(ba)^n=[a,b]^{{\tfrac 12n(n-1)}+1}$ holds with common value 1, then 4 divides $n$ and $\tfrac 12n(n-1)+1$ is odd, contrary to $[a,b]\neq 1.$ If the common value is not 1, then $\tfrac 12n(n-1)+1$ must be odd, i.e. $n=0$ or $1$ mod 4, but then $(ba)^n$ has order 1 or 4 respectively, contrary to $[a,b]^2=1.$ We conclude no $g\in G-Z(G)$ has order 4.
Then there must be $x\in Z(G)$ of order 4, since $G$ is not elementary abelian. If $g\in G-Z(G)$ is an involution, then $gx\in G-Z(G)$ has order 4, a contradiction. Hence in fact any $g\in G-Z(G)$ has order divisible by 8. Now if $g\in G-Z(G)$ is arbitrary and again we write $g=ba$ where $a$ and $b$ do not commute, then $(ba)^n=[a,b]^{\tfrac 12n(n-1)+1}$ implies $g^{2n}=1$ and so now 4 divides $n$ and again $\tfrac 12n(n-1)+1$ is odd and so $G^\prime\subseteq\langle g\rangle.$ As in the odd $p$ case, now $G$ is a Dedekind group, and as $G$ is not abelian, $Q_8$ is a subgroup of $G,$ a contradiction since $A(Q_8)$ does not hold.
Given the $p$-groups case, here is a proof that finite $G$ with $A(G)$ holding, is either abelian or of the form $G_k\times H$ where $G_k$ for each $k\ge 1$ is the group in the question and $H$ is abelian of order prime to 6. Write $\mathcal P$ for the set of isomorphism types of such groups.
I'll need that $A(G)$ does not hold for the following groups (all $\notin\mathcal P$):
- $G$ nonabelian of order 12 or 18 with nontrivial center. This is by direct checking.
- $G$ the Frobenius group $F_{pr}=C_p\rtimes C_r$ where $r$ divides $p-1,$ except $F_{3,2}=S_3=G_1$ To see this, let $x$ and $g$ be generators of $C_p$ and $C_r$ and write $b=g^2x,a=x^{-1}g^{-1}$ for some $u.$ Then $a$ and $b$ do not commute, $ba=g,$ and $a$ is not in $C_p.$ If $b$ is also not in $C_p$ then $a^r=b^r=1.$ Then $[a,b]=b^na^n\in C_p$ so $g^n=1$ in $F/C_p,$ so $r$ divides $n$ and so $a^n=b^n=1$ contradicting $[a,b]\neq 1.$ Hence $g^2\in C_p,$ so $r=2$ and $F$ is the dihedral group $D_{2p}.$ Then again pick any two reflections $s\neq t,$ then if $[s,t]=t^ns^n$ we have $(st)^2=ts,$ i.e. $(ts)^3=1$ as required.
By the $p$-groups case, we know that $A(G)$ holds implies $G$ has abelian Sylow subgroups. I next want to show that if $G/Z\in\mathcal P$ where $Z$ is central in $G,$ and if $A(G)$ holds, then $G\in\mathcal P.$ This is clear if $G/Z$ is abelian, because then $G$ is nilpotent and has abelian Sylow subgroups, so $G$ is abelian. So assume $G/Z=G_k\times H$ where 6 does not divide $H.$ By induction, we can assume $|Z|=p$ is prime. We claim $p\notin{2,3}.$ Otherwise the preimage of $G_1=S_3\subseteq G_k$ in $G$ must satisfy property $A,$ and is a nonabelian group of order 12, respectively 18, with nontrivial centre, for which $A$ does not hold by (1) above. Thus $p>3,$ so $G_k$ is the Hall $(2,3)$-subgroup of $G$ and $G_k\mathrm{char}G_kZ \vartriangleleft G$ If $H\vartriangleleft G$ is the Hall $(2,3)$-complement, then by the above $H$ is abelian, and $G=G_k\times H\in\mathcal P$ as desired.
Now let $G$ be minimal such that $A(G)$ holds and $G\notin\mathcal P.$ First, $G$ must be solvable. For let $P$ be a Sylow $2$-subgroup of $G.$ Groups in $\mathcal P$ have $P\subseteq Z(N_G(P)),$ so either $P\vartriangleleft G$ or $N_G(P)\in\mathcal P,$ which implies by Burnside's theorem that $P$ has a normal complement. Hence $G$ is solvable by the odd order theorem.
Also by the above, $Z(G)=1.$ Let $V$ be a minimal normal subgroup of $G$ and let $p$ be the unique prime dividing $|V|.$ If $x\in V$ and $y\in G$ is of order prime to $p$ then $[x,y]=y^nx^n$ for some $n\in\mathbb Z,$ but $[x,y]\in V$ so $y^n=1$ and $y$ normalizes $\langle x\rangle.$ We conclude $\langle x\rangle\vartriangleleft G,$ so $V=\langle x\rangle$ and has prime order $p.$
Since $Z(G)=1,$ and $G$ has abelian Sylow $p$-subgroups, some $y$ of $p^\prime$-order does not centralize $V.$ Then if $H=\langle x,y\rangle=V\rtimes\langle y\rangle,$ $\bar H=H/Z(H)$ is the Frobenius group $F_{pr}$ for $r$ the order of $\bar y.$ Since $A(\bar H)$ holds, by (2) above, $p=3.$ Thus $|V|=3$ and $|G:C_G(V)|=2.$ Notice also if $Q$ is a Sylow $2$-subgroup of $G$ then $A(VQ)$ holds, and this requires $|Q|=2$ since otherwise we could take $R\vartriangleleft Q$ with $|C_Q(V):R|=2$ and then $A(VQ/R)$ does not hold by (1) above, a contradiction. Hence $H=C_G(V)$ is abelian of odd order and $G=H\rtimes Q.$ Finally write $H=KL$ where $K$ and $L$ are the Sylow $3$-subgroup and the Hall $3$-complement of $H$ respectively. Since $LQ<G,$ $LQ\in\mathcal P$ so $Q$ centralizes $L.$ Finally, as above, if $x\in K$ and $y$ is an element of order 2, $[x,y]\in K$ so $[x,y]=y^nx^n$ requires $n$ even and $[x,y]=x^n,$ and we must have $[x,y]=x^{-1}$ since $Z(G)=1.$ Hence $G\in\mathcal P,$ a contradiction as required.
I haven't checked carefully, but I think this all also works for $B(G);$ if so then $A(G)$ and $B(G)$ are equivalent (i.e. both equivalent to the classification above).
