Let $G$ be a $p$group, $\{e\}\not= H\subseteq G$ be a subgroup of $G$ such that $G' = H'$. Is it true that $c(G) = c(H)$, where $c(\cdot)$ denotes the nilpotency class of a group?

2$\begingroup$ I suppose you assume $H$ to be nontrivial, otherwise $G = C_p$ for $p$ prime works as a counterexample. $\endgroup$– CarlFredrik Nyberg BroddaSep 20, 2023 at 8:07

1$\begingroup$ Clearly $c(H)=1$ implies $c(G)=1$. Actually, it is also true that $c(H)=2$ implies $c(G)=2$. This is a consequence of the HallWitt identity $[[a,b^{1}],c]^b \cdot [[b,c^{1}],a]^c \cdot [[c,a^{1}],b]^a = 1$, rewritten in the form $[[h_1,h_2],g]^{h_2^{1}} \cdot [[h_2^{1},g^{1}],h_1]^c \cdot [[g,h_1^{1}],h_2^{1}]^{h_1} = 1$. If $H$ is 2step nilpotent and $H'=G'$, both $[h_2^{1},g^{1}]$ and $[g,h_1^{1}]$ are in $H'$ and hence both righthand double brackets are trivial, so eventually $G$ centralizes $H'=G'$. $\endgroup$– YCorSep 20, 2023 at 8:33

1$\begingroup$ Finally this argument can be adapted to the general case, so I wrote an answer. $\endgroup$– YCorSep 20, 2023 at 9:01
1 Answer
This is true in an arbitrary nilpotent group.
Namely, for a nilpotent group $G$ and subgroup $H$ with $G'=H'$, let us check that $G^{(i)}=H^{(i)}$ for all $i\ge 2$ (lower central series. In particular, they have the same class (with the exception of $c(G)=1$, $c(H)=0$, i.e. $G$ nontrivial abelian $H$ trivial).
The case $i=2$ is the assumption $G'=H'$.
We use in general the HallWitt identity. $$[[a,b^{1}],c]^b \cdot [[b,c^{1}],a]^c \cdot [[c,a^{1}],b]^a = 1$$ (with conventions $x^y=y^{1}xy$ and $[x,y]=x^{1}y^{1}xy$).
Suppose that $G^{(j)}=H^{(j)}$ for all $2\le j\le i$. For $g\in G$, $h\in H$ and $s\in H^{(i1)}$, this formula shows that $[[h,s],g]$ belongs to $$[[G,H],H^{(i1)}][[H^{(i1)},G],H].$$ We have $$[[H^{(i1)},G],H]=[[G^{(i1)},G],H]=[G^{(i)},H]=[H^{(i)},H]=H^{(i+1)}.$$ Also $[[G,H],H^{(i1)}]=[[H,H],H^{(i1)}]$ is contained, again using the HallWitt identity, in $[[H^{(i1)},H],H]=H^{(i+1)}$. Hence $[[h,s],g]\in H^{(i+1)}$. Since such elements $[h,s]$ generate $H^{(i)}=G^{(i)}$, we deduce that $$G^{(i+1)}=[G,G^{(i)}]\subseteq [H,H^{(i)}]=H^{(i+1)}.$$
(By the way, this argument also works in the context of Lie rings.)

$\begingroup$ Thank you! mathoverflow.net/questions/454950/… is another question. $\endgroup$– solver6Sep 20, 2023 at 10:03

1$\begingroup$ You could also phrase your proof in terms of the three subgroups lemma (which follows from the HallWitt identity): Let $N$ be a normal subgroup of $G$ and $A,B,C \leq G$. If two among the subgroups $[A,B,C]$, $[B,C,A]$, $[C,A,B]$ are contained in $N$, then all of them are. ${}{}$ By the way, the problem appears in exercise 2.10 in Chap 4 of Suzuki, Group Theory II. Also in Corollary 1 of "Some sufficient conditions for a group to be nilpotent, P. Hall, Illinois J. Math. 2(4B): 787801 (December 1958). DOI: 10.1215/ijm/1255448649". $\endgroup$ Sep 20, 2023 at 10:03