Transitive permutation groups which all of their proper subgroups are intransitive

Question

Let $G$ be a transitive permutation group on a finite set $\Omega$. It is clear that if $G$ is regular, then every proper subgroup of $G$ is intransitive. Is there any other class of groups with this property? I mean that which transitive groups has no proper transitive subgroup?

Answer 1

Consider the symmetric group $S$ on $n$ symbols in its action on the $k$-subsets of $\{1,\ldots,n\}$. If $k\ne2,4$ and $n=2k+1$, then the only transitive subgroups of $S$ are $S$ itself and the alternating group. Hence in this case the alternating group acts transitively but all its proper subgroups are intransitive. See my "More Odd Graph Theory", Discrete Math. 32 (1980) 205-207. The basic idea is that a group that is transitive on $k$-subsets must be $(k-1)$-transitive and in most cases this leaves with only the symmetric and alternating groups.

Answer 2

As Geoff Robinson has already written in his comment, you are asking for minimally transitive permutation groups. Specifically you ask whether there is any class of such groups other than the regular permutation groups. The answer to this is a clear yes. -- For example there are up to conjugacy only $51$ regular permutation groups of degree $32$, but there is a total of $11605$ conjugacy classes of minimally transitive permutation groups of degree $32$. See Page 3 of the paper The Transitive Permutation Groups of Degree 32 by Cannon and Holt which reports about the determination of all $2801324$ conjugacy classes of transitive permutation groups of degree $32$.

Answer 3

Whenever one has a vertex-transitive graph which is not a Cayley graph, one has an example of such a group (maybe more than one). E.g. both $A_5$ and $5:4$ act vertex-transitively on the Petersen graph, and are minimal transitive subgroups of $S_{10}$. See e.g. the paper by B.D.McKay and C.E.Praeger.

Answer 4

As was implicit in my comment every transitive permutation group contains such a subgroup. I single out the case that $\Omega$ has prime power cardinality in my comment ( in which case any minimal transitive subgroup of ${\rm Sym}(\Omega)$ is a $p$-group, using Sylow's theorem). Just one further remark: in the case $|\Omega| = p^{n}$ for a prime $p$ and positive integer $n$, a transitive $p$-subgroup $G$ of ${\rm Sym}(\Omega)$ is a minimal transitive subgroup if and only if each point-stabilizer $G_{\omega}$ is contained in the Frattini subgroup $\Phi(G)$.

For if $G$ has a transitive proper subgroup $H$, then we have $HG_{\omega} = G$, so certainly $G_{\omega} \not \leq \Phi(G)$ as $G = \langle H, G_{\omega} \rangle \neq H$.

On the other hand, if $G$ is minimal transitive, and $M$ is a maximal subgroup of $G$, then $MG_{\omega} \neq G$ (otherwise $M$ would be transitive), so that $MG_{\omega} = M$ by maximality ( for $M \lhd G$, so that $MG_{\omega}$ is a subgroup of $G$). Hence $G_{\omega} \leq M$, so that $G_{\omega} \leq \Phi(G),$ as $M$ was an arbitrary maximal subgroup of $G$.

Later note: notice then that when $G$ is a finite $p$-group, there is a bijection between faithful transitive permutation representations of $G$ in which no proper subgroup is transitive, and $G$-conjugacy classes of subgroups $X \leq \Phi(G)$ such that $X \cap Z(G) = 1.$

For if $X$ is such a subgroup, then ${\rm core}_{G}(X) = 1$ ( otherwise, the core meets $Z(G)$ non-trivially), so the (transitive) permutation action of $G$ on the (say right) cosets of $X$ in $G$ is faithful. Each point stabilizer in this action is a $G$-conjugate of $X$, so is contained in $\Phi(G) \lhd G$ as $X \lhd G$. Hence by the above remark, $G$ is a minimal transitive permutation group on the cosets of $X$.

On the other hand, if $G$ has a faithful minimal transitive action on the right cosets of its subgroup $Y$, then $Y \leq \Phi(G)$ as $Y$ is a point stabilizer. Also $Y$ is core-free in $G$ since the permutation action is faithful, so we certainly have $Y \cap Z(G) = 1$.

Answer 5

Here's a description in the case when $|\Omega|$ is a prime power. As mentioned by Geoff Robinson, this case boils down to the case when we have an acting $p$-group.

Fix a prime $p$. Let $G$ be a $p$-group and $A=G/\Phi$ is its maximal $p$-elementary abelian quotient (so $\Phi$ is the Frattini subgroup), and $H$ a subgroup of $G$.

That $G$ is faithful on $G/H$ means that $H\cap Z(G)=1$ (standard).

That every proper subgroup is intransitive means that $H$ is contained in the Frattini subgroup $\Phi$.

Proof: (This is already in Geoff Robinson's post.) Indeed, if $H$ is not contained in $\Phi$, let $B$ be its projection on $A$, and write $A=B\oplus C$, and let $L$ be the inverse image of $C$ in $G$; this is a normal subgroup of $G$, so $LH$ is a subgroup of $G$. Since the projection of $LH$ on $A$ is all of $A$, we have $LH=G$. Conversely, if $H$ is contained in $\Phi$ and $M\subset G$ is transitive on $G/H$, then $MH=G$, which, in projection to $A$, implies that the projection of $M$ on $A$ is equal to $A$, which in turn implies that $M=A$. $\Box$

To get examples, we need groups $G$ for which the Frattini subgroup $\Phi$ containains elements of order $p$ that are not central in $G$. This is clearly impossible if $\Phi$ is central in $G$, which excludes examples of order $p^k$ for $k\le 3$. For $k=4$, it gives examples for $p\ge 3$: denoting by $C_p$ the cyclic group the wreath product $C_p\wr C_p$ has order $p^4$, its center is cyclic of order $p$ while its Frattini subgroup is isomorphic to $C_p^2$. On the other hand, it yields no example of order 16, that is, in every group of order 16, every element of order 2 in the Frattini subgroup is central (I did a case-by-case check but without a computer verification, please tell me if you check it's correct or not). Anyway there are examples of order 32, as already mentioned (for instance, the semidirect product $C_2^3\rtimes C_4$ where $C_4$ acts by a unipotent Jordan matrix; in this case, $H$ can be chosen to be $(\{0\}\times C_2\times \{0\})\rtimes C_2$ which defines an action of a set of cardinal 8).

Answer 6

A very repeatable construction to yield such permutation groups is to let a group act on the cosets of a small subgroup.
The idea is that if $H$ is a subgroup of $G$, a subgroup $T \leq G$ acts transitively on the cosets of $H$ if and only if $HT = G$. If $H$ is small, $T$ is very restricted (beginning, for example, with $[G:T] \leq |H|$) and it's easy, in many cases, to prove $T$ must be the whole $G$. If $G$ is regular, this corresponds to choosing $H = 1$, which forces $T=G$ without further discussion.