Subgroups of finite solvable groups with paradoxical properties

Question

For which pairs of finite solvable groups H, G, is the following true: H embeds in two ways into G, say as H1 and H2, where H1 is maximal in G and H2 is not? Are there any such pairs?

Some comments on the question.

1) Terminology: when H embeds in G maximally and non-maximally, call H a paradoxical subgroup of G, and say that G has H as a paradoxical subgroup.

2) For finite groups in general, as opposed to finite solvable, there are such things as paradoxical subgroups, it's not a vacuous concept. If H is a non-abelian simple group, and G is the direct product H x H, take H1 to be the diagonal subgroup, elements of G of the form (x,x), x in H, and take H2 to be a component subgroup, elements of G of the form (x,1), x in H. Then H1 and H2 are both isomorphic to H, H1 is maximal in G, and H2 isn't. I owe this example to Yair Glasner.

3) By Sylow's theorems, a p-group is never a paradoxical subgroup of any group.

4) A standard theorem on super-solvable groups, (that the maximal subgroups have prime index,) implies that supersolvabe groups have no paradoxical subgroups.

5) An important special case is when H1and H2 are both core-free, i e each one contains only the trivial group as a normal subgroup. in this case G has two faithful permutation representations of the same degree (namely the index of H in G), with one representation being primitive and the other imprimitive. Is this case any easier?

Answer 1

One can obtain a little more in Geoff's situation, where $O_p(G) > 1$. Write $U = O_p(G)$ and $U_i = U \cap H_i$, so $U_i = O_p(H_i)$, as Geoff showed. In particular, $|U_1| = |U_2|$. Note that the $U_i$ are proper in $U$. I claim that $U = U_1U_2$.

First, observe that $U_1 \triangleleft G$ since $N_H(U_1)$ contains $H_1$ properly because $N_U(U_1) > U_1$. Since $H_1$ is maximal, $U_1 \triangleleft G$, as claimed. Next, $U/U_1$ is $G$-chief by the maximality of $H_1$, and in particular, $U/U_1$ is abelian. Since $G = UH_2$, it follows that $H_2$ acts irreducibly on $U/U_1$. Now $H_2$ normalizes $U_2$, so it normalizes $U_1U_2$. Since $U_1 \subseteq U_1U_2 \subseteq U$, we have either $U_1U_2 = U_1$ or $U_1U_2 = U$.

Since $|U_1| = |U_2|$, the first possibility yields $U_1 = U_2$, and this is a normal subgroup invariant under an isomorphism from $H_1$ to $H_2$, so we have that the $U_i$ are trivial in this case, and $U$ is minimal normal in $G$. Since $G = UH_2$, this implies that $H_2$ is maximal, contrary to assumption. We thus have $U_1U_2 = U$, as claimed. In particular, this yields $G = H_1H_2$.

But where do we go from there?

---

OK, now I know where to go. I can finish the proof in the case where $O_p(G)$ is nontrivial.

Assume $G$ is a minimal counterexample. Let $D_2 = H_1 \cap H_2$ and let $D_1$ be the image of $D_2$ under some isomorphism from $H_2$ onto $H_1$. Thus $D_1$ and $D_2$ are isomorphic subgroups of $H_1$ with $p$-power index equal to $|G:H_2|$. I claim that $D_1$ is maximal in $H_1$ but $D_2$ is not, and also $O_p(H_1) = U_1 > 1$. Since $G$ is a minimal counterexample and $H_1 < G$, this will be contradict the minimality of $G$.

First, to see that $D_2$ is not maximal in $H_1$, choose a subgroup $X$ with $H_2 < X < G$. Since $H_1H_2 = G$, we deduce that $D_2 = H_1 \cap H_2 < H_1 \cap X < H_1$, as wanted.

Next, to show that $D_1$ is maximal in $H_1$, it suffices via the isomorphism to show that $D_2$ is maximal in $H_2$. Note that $H_1 = U_1D_2$ by Dedekind's lemma. Suppose $D_2 < Y < H_2$. Recall that $U_1 \triangleleft G$, so $H_1Y = U_1D_2Y = U_1Y$ is a subgroup. Also, $|H_1Y:H_1| = |Y:D_2|$, and it follows that $H_1 < H_1Y < G$, contradicting the maximality of $H_1$.

Now we need a proof (or counterexample) in the case where $O_p(G) = 1$.

---

My argument proves that if $H$ is paradoxical in solvable $G$ with $p$-power index, then $O_p(G) \not\subseteq H$. I did $not$ actually prove that $O_p(G) = 1$, but we do obtain this conclusion in a minimal counterexample.

Robinson has used my contribution and some deep theory to completely solve the problem for groups of odd order. (I am impressed.) We can also obtain some easier consequences. For example:

THEOREM. If H is supersolvable, then it cannot be paradoxical in a solvable group.

PROOF. As usual in a minimal counterexample, there is no nonidentity normal subgroup of $G$ contained in $H_1 \cap H_2$ and invariant under an isomorphism from $H_1$ to $H_2$. Now as $H_1$ is supersolvable, it has a nontrivial normal Sylow $q$-subgroup $Q$, and we know $q \ne p$, so $Q$ is Sylow in $G$. If $Q$ is normal in $G$ it is also Sylow in $H_2$ and is invariant under an isomorphism from $H_1$ to $H_2$, a contradiction. Otherwise, $H_1$ is the full normalizer of $Q$. A Sylow $q$-subgroup of $H_2$ is conjugate to $Q$ so its normalizer in $G$ is conjugate to $H_1$ and contains $H_2$. It follows that $H_2$ is maximal.

Answer 2

Re-edited in view of Derek Holt's comment : Some suggested directions- not a complete answer : Suppose we have $N \lhd G$ with $N \leq H_{1} \cap H_{2},$ and that there is an isomorphism $\alpha : H_{1} \to H_{2}$ which leaves $N$ invariant. Then $\alpha$ induces a well-defined isomorphism from $H_{1}/N$ to $H_{2}/N,$ and we may pass to $G/N.$ Hence we assume that there is no non-trivial choice of such an $N$ possible.

Let $[G:H_{1}] = p^{a}$ for some prime $p$ and integer $a.$ We note the general fact that if $M$ is a subgroup of the finite group $G,$ and $[G:M]$ is a power of $p,$ then $O_{p}(M) \leq O_{p}(G).$ It suffices to prove that $O_{p}(M) \leq P$ whenever $P \in {\rm Syl}_{p}(G).$ But $|PM| = \frac{|P||M|}{|P \cap M|}$, and this is divisible both by $|G|_{p^{\prime}}$ and by $|P|,$ so that $PM = G$ and $P \cap M \in {\rm Syl}_{p}(M)$. Hence $O_{p}(M) \leq P \cap M,$ as claimed. Hence we have $O_{p}(H_{i}) \leq O_{p}(G)$ for each $i.$ Note that $|O_{p}(H_{1})| = |O_{p}(H_{2})|.$ Suppose that $O_{p}(G) \leq H_{1}.$ Then $O_{p}(H_{2}) \leq O_{p}(G) \leq O_{p}(H_{1}) \leq O_{p}(G) $ and $O_{p}(H_{1}) = O_{p}(H_{2}) = O_{p}(G),$ which is then invariant under any isomorphism from $H_{1}$ to $H_{2}.$ Our assumptions then imply that $O_{p}(G) = 1.$ (Likewise, we can't have $O_{p}(G) \leq H_{2}$ if $O_{p}(G) \neq 1$- the maximality of $H_{1}$ is not essential in the preceding argument). Hence, if $O_{p}(G) \neq 1,$ then $G = O_{p}(G)H_{1}$ and $$|G| = \frac{|O_{p}(G)||H_{1}|}{|O_{p}(H_{1})|} = \frac{|O_{p}(G)||H_{2}|}{|O_{p}(H_{2})|},$$ so $G= O_{p}(G)H_{2}.$

Comments on the case $O_{p}(G) = 1$ : In this case, we now know that $O_{p}(H_{i}) = 1$ for each $i.$ Notice that $H_{1}$ and $H_{2}$ each contain a Hall $p^{\prime}$-subgroup of $G,$ and we may suppose that $H_{1}$ and $H_{2}$ have a common Hall $p^{\prime}$-subgroup, say $X$, possibly after replacing $H_{2}$ by a conjugate. We again may suppose that no non-trivial normal subgroup of $G$ which is contained in $H_{1} \cap H_{2}$ is invariant under any isomorphism from $H_{1}$ to $H_{2}.$ It follows that $O_{p^{\prime}}(H_{i})$ strictly contains $O_{p^{\prime}}(G)$ for each $i$ and that $F(H_{i})$ strictly contains $F(G)$ for each $i$ (in fact, the latter works prime by prime for $O_{q}(H_{i})$ for each $i ).$

Note that $F(H_{1})F(H_{2}) \leq F(X),$ so that $F(H_{1})F(H_{2})$ is, in particular, nilpotent.

Let $q$ be a prime for which $O_{q}(G)$ is non trivial. Then $O_{q}(H_{1}) > O_{q}(G)$ and $O_{q}(H_{1}) \leq X \leq H_{2}.$ Set $U = O_{q}(H_{1}).$ Then $H_{1} = N_{G}(U)$ as $U\not \lhd G$ and $H_{1}$ is maximal.

Now $$O_{q^{\prime}}(H_{1} \cap H_{2}) = O_{q^{\prime}}(N_{H_{2}}(U)) \leq O_{q^{\prime}}(H_{2}).$$

We claim that $O_{q^{\prime}}(H_{1})$ must have order divisible by $p$ if it is non-trivial. For otherwise, $ O_{q^{\prime}}(H_{1}) \leq X \leq H_{1} \cap H_{2}$ and $O_{q^{\prime}}(H_{1}) \leq O_{q^{\prime}}(H_{1} \cap H_{2}) \leq O_{q^{\prime}}(H_{2}),$ so $O_{q^{\prime}}(H_{1}) = O_{q^{\prime}}(H_{2}).$ But then $O_{q^{\prime}}(F(H_{1})) = O_{q^{\prime}}(F(H_{2}))$ and these are both non-trivial, a possibility we have excluded, as it leads to $H_{2} \leq H_{1}.$ This leaves two possibilities to consider. Firstly, it may be that $F(H_{1})$ is a $q$-group for some prime $q$. In that case, $F(G)$ is a $q$-group. We claim that $q<5$ in this case: for otherwise, for $Q \in {\rm Syl}_{q}(X),$ we have $ZJ(Q) \lhd G,$ and $ZJ(Q)$ is a normal subgroup of $G$ which is invariant under any isomorphism from $H_{1}$ to $H_{2}$.

The other possibility is that whenever $O_{q}(G) \neq 1$, $O_{q^{\prime}}(H_{1})$ has order divisible by $p,$ in which case, $O_{q^{\prime}}(G)$ must have order divisible by $p,$ as $O_{q^{\prime}}(H_{1}) \leq O_{q^{\prime}}(G).$

PROOF THAT THIS CAN'T HAPPEN IN GROUPS Of ODD ORDER: Let $G$ be a finite (solvable) group of odd order. We will prove that if $H_{1}$ is a maximal subgroup of $G$ and $H_{2}$ is a subgroup of $G$ isomorphic to $H_{1},$ then $H_{2}$ is also maximal. By previous work on this problem by myself and Marty Isaacs, we may suppose that $O_{p}(G) = 1,$ where $[G:H_{1}] = p^{a},$ where $p$ is prime and $a$ is a positive integer. As before, if $\alpha :H_{1} \to H_{2}$ is an isomorphism, we may suppose that nor normal subgroup of $G$ contained in $H_{1}$ is $\alpha$-invariant (note that,if it were, it would be contained in $H_{2}$ also).

Now if $X$ is a finite (solvable) group of odd order, and $O_{p^{\prime}}(X) =1,$ then $ZJ(P) \lhd X,$ where $J$ denotes the Thompson subgroup. This is a Theorem of Glauberman. ZJ(P) is even characteristic, as Glauberman observed. One way to see this is to note that $ZJ(P)$ is the intersection of all Abelian $p$-subgroups of $X$ of maximal order. This follows easily as $J(P^{g}) = J^(P)^{g},$ and so $ZJ(P)$ does not depend on the particular Sylow $p$-subgroup $P$ chosen. It follows, that $ZJ(P)$ centralizes, hence is contained in, each Abelian $p$-subgroup of $X$ of maximal order. A nilpotent injector $I$ of a finite solvable group $X$ is a maximal nilpotent subgroup of $X$ which contains $F(X).$ A nilpotent injector is unique up to conjugacy in $X,$ and for each prime $q$, the subgroup $O_{q}(I)$ is a Sylow $q$-subgroup of $C_{G}(O_{q^{\prime}}(X)).$ We claim that when $X$ has odd order, and I is a nilpotent injector of $X,$ then $ZJ(I) \lhd X.$ It suffices to prove that $ZJ(O_{q}(I)) \lhd X$ for each prime $q.$ (Here, $J(I)$ denotes the subgroup of $I$ generated by its Abelian subgroups of maximal order). Let $C = C_{X}(O_{q^{\prime}}(X)).$ Then $O_{q}(I) \in {\rm Syl}_{q}(C)$ and so $C = O_{q^{\prime}}(C)N_{C}(ZJ(O_{q}(I))$ by Glauberman's theorem. But $O_{q^{\prime}}(C) \leq O_{q^{\prime}}(X),$ so that $ZJ(O_{q}(I)) \lhd C$. And it is even characteristic, because $O_{q^{\prime}}(C) \times ZJ(O_{q}(I))$ is characteristic. Since $C \lhd X,$ we have $ZJ(O_{q}(I)) \lhd X,$ as required. Hence $ZJ(I) \lhd X.$

Note that $ZJ(I) \leq F(X),$ so that $ZJ(I) \leq T$ for every nilpotent injector $T$ of $X.$ Since $T = I^{x}$ for some $x,$ we see that $ZJ(I) = ZJ(T)$ is independent of the particular nilpotent injector $I$ of $X$ which is chosen.

Now let us turn to our solvable group $G$ of odd order with its maximal subgroup $H_{1}$ and isomorphic subgroup $H_{2}$, both of index a power of $p,$ where $O_{p}(G) = 1.$

We may suppose that $H_{1}$ and $H_{2}$ contain a common Hall $p^{\prime}$-subgroup $M$ of $G,$ possibly after replacing $H_{2}$ by a conjugate. Now $F(H_{1})F(H_{2}) \leq M,$ so that $F(H_{1})F(H_{2}) \leq F(M).$ Let $I$ be a nilpotent injector of $M$ containing $F(M).$ Then $I$ is a nilpotent injector of $H_{1}$ and of $H_{2},$ since it contains $F(H_{1})$ and $F(H_{2})$ and $O_{p}(H_{i}) = 1$ for each $I.$

Also, $I$ is a nilpotent injector of $G,$ since $F(G)$ is a $p^{\prime}$-group. Thus $ZJ(I) \lhd G.$ How let $\alpha :H_{1} \to H_{2}$ be an isomorphism. Then $T = I\alpha$ is a nilpotent injector of $H_{2},$ and also a nilpotent injector of $G.$ Now $ZJ(I)\alpha = ZJ(I\alpha) = ZJ(T) = ZJ(I).$ This contradicts our assumption that there was no non-trivial $\alpha$-invariant normal subgroup of $G$ contained in $H_{1}.$