Can $1\ne H\cap H^g\lhd H$ happen if $G$ is a primitive permutation group with stabiliser $H$?

Assume everything is finite.

Let $$G$$ be a primitive permutation group with point stabiliser $$G_\alpha$$ for some $$\alpha$$. For $$\beta\ne\alpha$$, by an arc stabiliser we mean $$G_{\alpha\beta}=G_\alpha\cap G_\beta$$ and an edge stabiliser we mean $$G_{\{\alpha,\beta\}}$$, the stabiliser of the set $$\{\alpha,\beta\}$$. Note that $$G_{\{\alpha,\beta\}}\ge G_{\alpha\beta}$$.

My question is: can an arc stabiliser $$G_{\alpha\beta}\ne 1$$ be normal in $$G_\alpha$$?

This is impossible if $$G_{\{\alpha,\beta\}}> G_{\alpha\beta}$$. In this case, there exists $$g\in G$$ such that $$\alpha^g=\beta$$ and $$\beta^g=\alpha$$. It follows that $$g\in N_G(G_{\alpha\beta})$$. Note that $$G_\alpha$$ is maximal in $$G$$ and $$G_{\alpha\beta}$$ cannot be normal in $$G$$ (otherwise $$G_{\alpha\beta}$$ will be in the kernel of this action), the normaliser $$N_G(G_{\alpha\beta})=G_\alpha$$. This gives a contradiction: $$g\in G_\alpha$$ but $$\alpha^g=\beta\ne \alpha$$.

I think there might exist an example in the case when $$G_{\{\alpha,\beta\}}=G_{\alpha\beta}$$. However, by a quick check with Magma there is no such primitive permutation group $$G$$ with $$|G|\le 300$$.

An equivalent statement is: Let $$G$$ be a group and $$H$$ a maximal and core-free subgroup of $$G$$. Is it possible that $$1\ne H\cap H^g\lhd H$$ for some $$g\notin H$$?

As shown in the comments there by Verret and Holt there is no such example for degree $$\le 4095$$. I also think the proof or an example, if any, will not be elementary (for example, applying O'Nan-Scott Theorem).

This was initially a MSE question I asked.

• @GeoffRobinson I think that remark is essentially equivalent to $G_{\{\alpha,\beta\} }> G_{\alpha\beta}$. In the language of permutation groups, this is saying that the orbit of $\alpha^g$ under $G_\alpha$ is not self-paired. I conjecture that the answer to the question is no, but that the proof will require CFSG. (The question reminds me of the Sims Conjecture that, for a primitive group $G$, $|G_\alpha|$ is bounded as a function of $|\beta^{G_\alpha}|$ for any $\beta \ne \alpha$, which was evenutally proved using CFSG.) Sep 23 '20 at 16:01
• @DerekHolt : Thanks. I agree that it is likely to be true that there is no such example (but I am not totally sure- for almost simple groups it seems very likely, certainly). Sep 24 '20 at 8:33
• I am reminded of the following generalization by Wielandt of a famous theorem of Frobenius. The Theorem of Wielandt asserts that if a finite group $G$ has a subgroup $H$, such that there is a normal subgroup $H_{0} \lhd H$ such that $H \cap H^{g} \leq H_{0}$ for all $g \in G \backslash H$, then there is $K \lhd G$ such that $G = KH$ and $K \cap H = H_{0}$. Sep 25 '20 at 20:21

An example was constructed by Pablo Spiga: https://arxiv.org/abs/2102.13614 "A generalization of Sims conjecture for finite primitive groups and two point stabilizers in primitive groups"

• Yes, I have also noticed that paper. Thanks for your comment. We then think this statement holds true for "most" primitive groups. For example, a series of papers by Konygin (references of Pablo Spiga's paper) handle almost all the cases when $G$ is of AS or PA type. Mar 3 '21 at 8:56

Sorry, this is not an answer, but rather an application of standard techniques of local analysis to obtain substantial structural information about $$H\cap H^g$$, in case an example exists. The techniques are inventions of John Thompson, George Glauberman, and Helmut Bender.

Proposition: Let $$G$$ be a primitive permutation group on a set $$\Omega$$, and let $$\alpha$$ and $$\beta$$ be distinct points in $$\Omega$$. If $$1\ne G_{\alpha\beta}\triangleleft G_\alpha$$, then $$G_{\alpha\beta}$$ has the following properties:

(a) $$F^*(G_{\alpha\beta})$$ is either a $$p$$-group for some prime $$p$$, or the direct product of nonabelian simple groups.

(b) If $$G_{\alpha\beta}$$ is solvable, then $$F^*(G_{\alpha\beta})$$ is a $$2$$-group or a $$3$$-group.

(c) If $$P$$ is a Sylow subgroup of $$G_{\alpha\beta}$$, then no nontrivial characteristic subgroup of $$P$$ is normal in $$G_{\alpha\beta}$$.

(d) If $$F^*(G_{\alpha\beta})$$ is a $$p$$-group, $$p>2$$, then $$G_{\alpha\beta}$$ is not $$p$$-stable.

(e) (Uses Odd Order Theorem) $$G_{\alpha\beta}$$ has even order.

Proof: First, part (c). Suppose that $$P$$ is Sylow in $$G_{\alpha\beta}$$, $$P_0\ne 1$$ is a characteristic subgroup of $$P$$, and $$P_0\triangleleft G_{\alpha\beta}$$. By the Frattini argument $$G_\alpha=G_{\alpha\beta}N_{G_\alpha}(P)\le N_{G_\alpha}(P_0)$$ so $$G_\alpha=N_G(P_0)$$ by maximality of $$G_\alpha$$. Hence $$N_{G_\beta}(P)\le N_{G_\beta}(P_0)=G_{\alpha\beta}$$. Since $$P$$ is Sylow in $$G_{\alpha\beta}$$, it is Sylow in $$G_\beta$$ and hence also in $$G_\alpha\cong G_\beta$$. Now $$G_\beta=G_\alpha^g$$ for some $$g\in G$$, so $$P^{gh}=P$$ for some $$h\in G_\beta$$, by Sylow's theorem. Then $$gh\in N_G(P)\le N_G(P_0)$$ so $$P_0\triangleleft G_\alpha^{gh}=G_\beta$$. Then $$P_0\triangleleft\langle G_\alpha,G_\beta\rangle=G$$, so $$P_0=1$$ as $$G_\alpha$$ contains no nontrivial normal subgroup of $$G$$. But $$P_0\ne1$$, contradicttion.

Next, part (a). If $$F(G_{\alpha\beta})=1$$, then $$F^*(G_{\alpha\beta})=E(G_{\alpha\beta})$$ and it is the direct product of nonabelian simple groups. So suppose $$B=O_p(G_{\alpha\beta})\ne 1$$ for some prime $$p$$. Then $$N_G(B)=G_\alpha$$. Let $$A=O^p(F^*(G_{\alpha\beta}))$$, the subgroup of $$F^*(G_{\alpha\beta})$$ generated by all its $$p'$$-elements. By the structure of generalized Fitting subgroups, $$[A,B]=1$$. If $$A=1$$, then $$F^*(G_{\alpha\beta})=B$$ and we are done, so assume that $$A\ne 1$$. Then $$N_G(A)=G_\alpha$$. Let $$a\in A$$ be an arbitrary $$p'$$-element and consider the action of $$\langle a\rangle\times B$$ on $$C=O_p(G_\beta)\triangleleft G_\beta$$. We have $$[\langle a\rangle,C_C(B)]\le [A,G_\alpha]\cap[G_\beta,C]\le A\cap C\le O_p(A)$$. But $$A$$ is the elementwise-commuting product of $$q$$-groups for various primes $$q\ne p$$, and quasisimple groups. Therefore $$[\langle a\rangle, C_C(B)]\le Z(A)$$ so $$[\langle a\rangle,C_C(B),\langle a\rangle]=1$$. As $$a$$ is a $$p'$$-element normalizing the $$p$$-group $$C_C(B)$$, a standard coprime action lemma implies that $$[\langle a\rangle,C_C(B)]=1$$.Then the Thompson $$A\times B$$ Lemma implies that $$[\langle a\rangle,C]=1$$. But $$a$$ was an arbitrary generator of $$A$$, so $$[A,C]=1$$. This implies in turn that $$C\le N_G(A)=G_\alpha$$. Since $$C$$ is a normal $$p$$-subgroup of $$G_\beta$$, $$C\le O_p(G_{\alpha\beta})=B$$. But $$B\le O_p(G_\alpha)\cong O_p(G_\beta)=C$$, so $$B=C$$. Since $$B\triangleleft G_\alpha$$ and $$C\triangleleft G_\beta$$, $$B\triangleleft G$$, a contradiction as $$B\le G_{\alpha}$$.

Now (d). If $$F^*(G_{\alpha\beta})$$ is a $$p$$-group, then a Sylow $$p$$-subgroup $$P$$ of $$G_{\alpha\beta}$$ is nontrivial. If also $$p>2$$ and $$G_{\alpha\beta}$$ is $$p$$-stable, then by Glauberman's $$Z(J)$$-Theorem, $$Z(J(P))\triangleleft G_{\alpha\beta}$$ for any Sylow $$p$$-subgroup $$P$$ of $$G_{\alpha\beta}$$. But $$Z(J(P))$$ is a nontrivial characteristic subgroup of $$P$$. This contradicts (c).

Now (b). By (a) and solvability, $$F^*(G_{\alpha\beta})$$ is a $$p$$-group for some prime $$p$$. Suppose $$p>2$$. By (d), $$G_{\alpha\beta}$$ is not $$p$$-stable, whence $$G_{\alpha\beta}$$ has a subquotient isomorphic to $$SL_2(p)$$. So $$SL_2(p)$$ is solvable and $$p=3$$.

Finally, (e). Suppose that $$G_{\alpha\beta}$$ has odd order, so it is solvable. By (b) and (d), $$F^*(G_{\alpha\beta})$$ is a $$3$$-group and $$G_{\alpha\beta}$$ is not $$3$$-stable. Hence $$G_{\alpha\beta}$$ has an $$SL_2(3)$$ subquotient, which is of even order, contradiction.

• These are very good criteriums on such $G_{\alpha\beta}$ (of course, if any). Thank you for your observations. Sep 27 '20 at 2:25
• If $G_{\alpha \beta}$ is solvable, I think you can get that either $G$ involves ${\rm Qd}(3)$ or $G$ involves $S_{4}$. In particular, a Hall $\{2,3\}$-subgroup of $G$ is neither $2$-closed nor $2$-nilpotent. This needs a Theorem of Stellmacher, as well as Glauberman's $ZJ$-theorem.( using condition c)). Oct 1 '20 at 20:55
• In the above comment, $G$ means $G_{\alpha \beta}.$ Oct 3 '20 at 10:28