For a new operation on a finite group of odd order giving a loop structure, when does this also gives a group

Question

For finite groups $G$ of odd order, as $x \mapsto x^2$ is bijection (but no automorphism in general) then, we can define for each $g \in G$ the element $x^{1/2}$ by requiring $(x^{1/2})^2 = x$. Then with the new operation $$ x\circ y := x^{1/2} y x^{1/2} $$ we get a power-associative loop satisfiying the relations $$ x\circ (y \circ (x \circ z)) = (x\circ (y\circ x))\circ z $$ and $$ (x\circ y)^{-1} = x^{-1} \circ y^{-1}. $$ If the first is satisfied, such loops are also called (left) Bol loops, and if both additional equations hold they are called (left) Bruck loops. These definitions go back to ideas of G. Glauberman: On loops of odd order. Now I want to know, under what conditions for $G$ to we indeed have associativity? As $1 \in G$ is also neutral for the new operation, and also every inverse is also an inverse in the new operation, the question is under what conditions is $(G, \circ)$ also a group?

Answer 1

As I mentioned in comments, I believe that H. Bender has used this operation in special situations(in particular for certain groups of odd order of nilpotence class 2) , and that his use of it appears in the book "Topics in Finite Groups" by T. Gagen ( published by LMS as I recall). I don't remember all details but here is a proof that when $G$ is a nilpotent group of (finite) odd order and nilpotence class 2, then we always have $x \circ y = y \circ x$.

Recall that if $G$ has nilpotence class 2, then we have $[xy,z] = [x,z][y,z]$ for all $x,y, z \in G$ as all commutators in $G$ are central. In particular, $[x,y]^{2} = [x^{2},y] = [x,y^{2}]$ for all $x,y \in G$. Also, we have always have $[x,y]^{-1} = [y,x]$.

Since when $G$ has odd order, we know that $x^{\frac{1}{2}}$ is an (integer) power of $x$, and likewise for $y$, we have both $[x^{\frac{1}{2}},y]^{2} = [x,y]$ and $[x,y^{\frac{1}{2}}]^{2} = [x,y]$, so we conclude that we always have $[x^{\frac{1}{2}},y] = [x,y^{\frac{1}{2}}] = [x,y]^{\frac{1}{2}}.$

Now from the definition of $x \circ y$, we may write $x \circ y = xy[y,x^{\frac{1}{2}}]$, and likewise, we may write $y \circ x = yx[x,y^{\frac{1}{2}}]$.

Now all commutators are central in $G$, so we now have $$(x \circ y)^{-1}(y \circ x) = [x^{\frac{1}{2}},y]y^{-1}x^{-1}(yx)[x,y^{\frac{1}{2}}] = [x^{\frac{1}{2}},y]^{2}[y,x] = [x,y][y,x] = 1,$$ so that $x \circ y = y \circ x$, as claimed.

Later edit: Actually, I think I can reconstruct the proof of associativity in this case: We have $x \circ (y \circ z) = x(y\circ z)[y \circ z,x]^{\frac{1}{2}}$ from above.

Again recalling that commutators are central in $G$, we may continue to obtain $$ x \circ (y \circ z) = x(yz)[z,y]^{\frac{1}{2}}[yz,x]^{\frac{1}{2}},$$ where we make use of the fact that $[[z,y^{\frac{1}{2}}],x] = 1$.

Hence $$x \circ (y \circ z) = x(yz)[z,y]^{\frac{1}{2}}[y,x]^{\frac{1}{2}}[z,x]^{\frac{1}{2}}$$.

Similarly (and making use of commutativity), we obtain $$(x \circ y) \circ z = z \circ (x \circ y) = z(xy)[y,x]^{\frac{1}{2}}[x,z]^{\frac{1}{2}}[y,z]^{\frac{1}{2}}$$.

Taking the product of the inverse of the first expression with the second expression ( and recalling that commutators are central), we obtain $(yz)^{-1}x^{-1}zxy [x,z][y,z] = [z,xy][xy,z] = 1$. Hence $\circ$ is associative.

Later edit: In general, given that $(G,\circ)$ is a group, it is in fact necessary that $G$ (with its original operation) should be nilpotent, though I make no attempt to bound the nilpotency class. We have seen that if $G$ has nilpotence class $2$ (or less) then $(G,\circ)$ is a group.

Suppose then that our $G$ of odd order is such that $(G,\circ)$ is a group ( but we make no further assumptions on $G$ with respect to its original operation). Then $(G,\circ)$ is in fact an Abelian group, as noted in comments for we are told that $(x \circ y)^{-1} = x^{-1}\circ y^{-1}$ (which can also be verified by direct computation), while we have $(x \circ y)^{-1} = y^{-1} \circ x^{-1}$ if $\circ$ is associative.

Thus we have $x^{\frac{1}{2}}yx^{\frac{1}{2}} = y^{\frac{1}{2}}xy^{\frac{1}{2}}$ for all $x,y \in G$. As M. Farrokhi D.G. notes in comments below, this is equivalent to requiring that $x^{\frac{1}{2}}y^{\frac{1}{2}}$ and $y^{\frac{1}{2}}x^{\frac{1}{2}}$ commute for all $x,y \in G$. Since squaring is a bijection on $G$ this is equivalent to $[xy,yx] = 1$ for all $x,y \in G$. Thus y and $x^{-1}yx$ commute for all $x,y \in G$. Thus $[[y,x],y] = 1= [[x,y],y] = 1$ for all $x,y \in G$. Thus $G$ is an Engel group. Engel groups are well-studied and finite Engel groups are known to be nilpotent (apparently credited to Zorn), though there are examples of infinite Engel groups which are not nilpotent. Hence $G$ is nilpotent with respect to its original group operation.

Even later edit: I will now address the question of nilpotence class, making use of a survey article by Gunnar Traustason, which is available online. The group $G$ is a $2$-Engel group, as seen above. Since we now know that $G$ is nilpotent, $G$ decomposes as a direct product of its Sylow $p$-subgroups, and clearly $(G,\circ)$ does too, in a way which respects the decomposition for $G$.

Hence it suffices to understand the situation when $G$ is a $p$-group for some odd prime $p$. It was in fact (at least implicitly) proved by W. Burnside that a $2$-Engel group of order prime to $3$ has nilpotency class $2$ ( and we have already seen that for a group $G$ of nilpotency class $2$, $(G,\circ)$ is indeed a group). It was proved somewhat later that a $3$-group which is a $2$-Engel group has nilpotency class at most $3$. Hence the only outstanding question at this point is whether there is a $3$-group $G$ of nilpotency class $3$ such that $(G,\circ)$ is a group. For we know that a group $G$ of odd order such that $(G,\circ)$ is a group is a direct product of a $3^{\prime}$-group which is nilpotent of class at most $2$ with a $3$-group which is nilpotent of class at most $3$.

Final edit: In fact, I think it already follows from Lemma 7 of the paper of Glauberman mentioned in the statement of the question that if $(G,\circ)$ is a (necessarily Abelian as remarked before) group, then $G$ must be nilpotent of class $2$.