Transitive subgroup of $S_p$ containing a $p$-cycle and a double transposition

Question

Let $p$ be a prime other than 5 or 7. Are $A_p$ and $S_p$ the only subgroups of $S_p$ that contains a $p$-cycle and a double transposition?

As for $p = 5$, the dihedral group $D_{10}$ contains a 5-cycle and a double transposition. For $p = 7$, the group $PSL_3(\mathbb{F}_2)$ (acting on the projective plane of order 2) contains a 7-cycle and a double transposition.

Answer 1

I believe that it was already known to Jordan that if $n \geq 8$ and $G$ is a primitive subgroup of $S_n$ containing a double transposition, then $G$ contains $A_n$. As a subgroup of $S_p$ containing a $p$-cycle is primitive, the answer to your question is ``yes".

Answer 2

There is a way to prove this via analogy with a proof approach which works for transpositions and 3-cycles.

For transpositions and 3-cycles on 4 or more points, the same proof (more or less) works easily to show that a primitive permutation group on $n$ points containing a transposition (resp., a 3-cycle) must be $S_{n}$ (resp., or $A_{n}$):
Define a relation on the indices by $i \sim j$ means "there is a transposition (resp., 3-cycle) in $G$ sending $i$ to $j$".
This relation is reflexive: Since $G$ is primitive, $G$ is transitive. $n \geq 4$ means that transpositions and 3-cycles have fixed points. The transitivity of $G$ allows a fixed point of a transposition or 3-cycle to be moved (by conjugation) to any desired index.
This relation is symmetric: If $\sigma$ is a transposition or a 3-cycle sending $i$ to $j$, then $\sigma^{-1}$ is, respectively, a transposition or 3-cycle sending $j$ to $i$.
This relation is transitive: This one takes more work. The cases where any two of $i$, $j$ and $k$ are equal are trivial, so assume they are distinct. Suppose $\alpha$ sends $i$ to $j$ and $\beta$ sends $j$ to $k$. If $\alpha$ and $\beta$ are both transpositions, then $< \alpha , \beta>$ acts as $S_{3}$ on {$i, j, k $}. Since this also contains the transposition $ (i,k) $, we are done. If $\alpha$ and $\beta$ are both 3-cycles, then $ < \alpha, \beta> $ acts as $A_{3}$, $A_{4}$ or $A_{5}$ on the union of the sets of indices moved by $\alpha$ and $\beta$. This alternating group contains all 3-cycles moving these points, so it contains the one sending $i$ to $k$.
So that relation is an equivalence relation. Moreover, from its definition, it is clear that it is a $G$-invariant equivalence relation on the indices. Then, since $G$ contains a transposition (resp., a 3-cycle), this equivalence relation has fewer than $n$ equivalence classes. Since $G$ is primitive, this means that all indices are equivalent.
In the case of transpositions, generating the whole $S_{n}$ is easy: since any two indices are switched by a transposition, $G$ contains all transpositions and thus $G = S_{n}$.
In the case of 3-cycles, it is slightly trickier: start with the 3-cycle $(1,2,a)$ which is in $G$. This generates $A_{3}$ on these 3 points. Now extend this subgroup by adjoining a 3-cycle which sends $1$ to an unmoved index. This results, as noted before, in $A_{4}$ or $A_{5}$. Again we extend this generating set by adjoining a 3-cycle which sends a moved index to an unmoved one, until all indices have been moved.
The claim is that the result of extending the group by adjoining the next 3-cycle is a new alternating group on all the moved points. If the next 3-cycle has only one unmoved point, this is immediate from the orbit-stabilizer theorem. If the next 3-cycle has 2 unmoved points, then the resulting group is transitive on $r \geq 6$ points and has a point stabilizer of order at least $\frac{ (r-2)!}{2}$. This means the extended group has order at least $\frac{ r (r-2)!}{2}$, so its index in $A_{r}$ is at most $r-1$. Since $r \geq 6$, $A_{r}$ has no subgroups of index less than $r-1$ except for itself. So the extended group is an alternating group, and $G$ has enough 3-cycles to generate $A_{n}$. Thus $G = A_{n}$ or $S_{n}$.

Now, just as moving from transpositions to 3-cycles introduced new hoops to jump through, moving from 3-cycles to double transpositions makes carrying out this idea even trickier. We now assume, as the problem specifies, that $n \geq 9$.
We define the same relation and want to prove it's an equivalence relation. Reflexivity only requires $n \geq 5$, so we're good. Symmetry is immediate.
Transitivity is more work: Suppose $\alpha$ is a double transposition which sends $i$ to $j$, and $\beta$ is a double transposition which sends $j$ to $k$. As before, the problem is trivial if any of $i$, $j$ and $k$ coincide. So assume $i$, $j$ and $k$ are distinct. The conjugate $\beta \alpha \beta$ is also a double transposition and it sends $i$ to $k$ unless $\beta$ also moves $i$. So write $\alpha = (i,j)(k,l)$ and $\beta = (j,k)(i,m)$. Here $i$, $j$, $k$ and $l$ are distinct by assumption, and $m$ is not $i$, $j$ or $k$. If $m = l$, then $\alpha \beta$ sends $i$ to $k$ ( and, in fact, $< \alpha, \beta >$ acts as $V$ on {$i ,j , k, l$} ). If $m$ is a fifth index, then $< \alpha, \beta >$ acts as the dihedral group of order 10 on {$i, j, k, l, m$} and this group has just enough double transpositions to send any index to any other. So transitivity is done.
As before, this relation is an equivalence relation. Since $G$ by assumption contains double transpositions, this equivalence relation, as before, makes all indices equivalent and there is a double transposition in $G$ switching any two desired indices.
The trickiest part of this proof is building up the set of double transpositions to generate $A_{n}$.
We start with the subgroup generated by a single double transposition, $(1,2)(i,j)$. This group has 2 orbits for its action on its set of moved points. The number of orbits for the action of the group we build on the set of moved points never needs to exceed 2, since we may adjoin our double transposition to switch two points in different nontrivial orbits and get at most one other orbit from the other transposition. In fact, continuing in this way as long as we have two orbits of moved points gives us, at each step, a group acting transitively on its set of moved points or a group acting with two orbits on its moved points, one of size 2 and the other (the large orbit) on which the group acts faithfully.
If the group we build is ever transitive on its set of moved points and there are still unmoved points, adjoin a double transposition which switches a moved point with an unmoved point. (This also keeps the number of orbits less than or equal to 2, with a large orbit and a 2-point orbit when there are 2 orbits on the moved points.)
Before going further, it is good to describe how to handle the extreme cases, where the group acts as $ (S_{r-2} \times S_{2}) \cap A_{r}$ or $A_{r}$ on its $r \geq 4$ moved points. If we have $A_{r}$, then if there are no unmoved points $r = n$ and $G$ contains $A_{n}$ as desired. If $r < n$, then adjoin a double transposition which switches a moved point with an unmoved point. If the two points of the other transposition are already moved, then the resulting group can only be $A_{r+1}$. If the two points of the other transposition are both unmoved, we likewise must have $ (S_{r+1} \times S_{2}) \cap A_{r+3}$. If one is moved and the other is unmoved, then the resulting group is transitive on $r+2$ points, consisting entirely of even permutations, and containing $A_{r}$ in a point stabilizer. Since $r \geq 4$, $r+2 \geq 6$ and, following the reasoning used before with 3-cycles, the group obtained must be $A_{r+2}$.
If the group acts as $ (S_{r-2} \times S_{2}) \cap A_{r}$ on $r > 4$ points (the inequality is strict to exclude the position we start from, with a lone double transposition), then adjoin a double transposition which switches a point in the large orbit and a point in the 2-point orbit. If the two points of the other transposition are both unmoved, we have $ (S_{r} \times S_{2}) \cap A_{r+2}$ as our new group. If both points of the other transposition are moved, then the maximality of $ (S_{r-2} \times S_{2}) \cap A_{r}$ in $A_{r}$ (here I am using the assumption that $r > 4$) implies that the new group must be $A_{r}$. If one point of the other transposition is moved and one is unmoved, then considering which 3-cycles are in the resulting group (remember that $r > 4$ so $ (S_{r-2} \times S_{2}) \cap A_{r}$ has 3-cycles in it) shows it must be $A_{r+1}$.
So if, at any point in this process, we obtain a group that acts as $A_{r}$ or $ (S_{r-2} \times S_{2}) \cap A_{r}$ on its $r > 4$ moved points, then the adjunction process will ultimately give us $A_{n}$ on our $n$ points when we have adjoined enough double transpositions. However, unlike the situation with 3-cycles, we don't settle into this situation so easily.

Applying the procedure to the subgroup generated by our first double transposition, $(1,2)(i,j)$, we adjoin a double transposition switching $2$ and $i$. The other transposition of this adjoined element switches two moved points, two unmoved points, or one moved point and one unmoved point. If it switches two moved points, it is $(2,i)(1,j)$ and the result is $V$ on {$1,2,i,j$}. If it switches one moved point and one unmoved point, the result is $D_{10}$ on 5 points. If it switches two unmoved points, the result is $D_{8}$ on the 4 points of the large orbit. The basic idea of the rest of the proof is this: continue all the cases of this adjunction procedure until they all lead to dead ends (via $ (S_{r-2} \times S_{2}) \cap A_{r}$ or $A_{r}$ moving more than 4 points, or what immediately follows this). (It is in working out these cases that the assumption that $n \geq 9$ winds up being fully used.)
One issue that description doesn't capture, however, is the fact that $V$ leads to another infinite family of groups: $V$ can be thought of as $ (S_{2} \wr S_{2}) \cap A_{4}$. $ (S_{2} \wr S_{t}) \cap A_{2t} $ is transitive on $2t$ points for any $t \geq 2$, and we adjoin a double transposition which switches a moved point (which we will call $1$) with an unmoved point (which we will call $2t+1$). However, the other transposition might be $(2,2t+2)$, in which case we have extended $ (S_{2} \wr S_{t}) \cap A_{2t}$ to $ (S_{2} \wr S_{t+1}) \cap A_{2t+2}$. To handle this, prove that $t > 4$ implies that that group is a maximal subgroup of $A_{2t}$. This will work (though it's written in a way that shows I haven't polished it):

Suppose $K$ is strictly intermediate between $ (S_{2} \wr S_{t}) \cap A_{2t}$ and $A_{2t}$.

1. Show that $K$ is doubly transitive. It's already given that $K$ has elements which send two indices in different pairs (thinking of $ (S_{2} \wr S_{t}) \cap A_{2t}$ as the stabilizer of a pairing in $A_{2t}$) to two indices in different pairs, and two paired indices to two paired indices. We only need to show that $K$ can send two paired indices to two indices in different pairs and vice versa. This is done by conjugating an element of $K$ outside $ (S_{2} \wr S_{t}) \cap A_{2t}$ appropriately.
2. A 2-point stabilizer in $K$ contains $ (S_{2} \wr S_{t-1}) \cap A_{2t-2}$, so $K$ is, in fact, triply transitive.
3. A 3-point stabilizer in $K$ contains $ (S_{2} \wr S_{t-2}) \cap A_{2t-4}$, so either it is transitive or it has a fixed point and an orbit of size $2t-4$. It can't be transitive because then $K$ would be quadruply transitive and contain double transpositions, so $K$ would contain all double transpositions, contradicting $K < A_{2t}$. So it has a fixed point and an orbit of size $2t-4$.
4. This means that $K$ preserves a $S(3,4,2t)$ Steiner system. To extend a set of 3 points to a set of 4, take the (pointwise) 3-point stabilizer, and adjoin the extra fixed point one gets outside the orbit of size $2t-4$.
5. We want to identify the points of this $S(3,4,2t)$ Steiner system with the elements of an affine space over $ \mathbb{Z} / (2) $. Equivalently, we want to identify an arbitrary chosen point with the zero vector and identify the points of the remaining $ S(2, 3, 2t-1) $ Steiner system with nonzero vectors in an $ \mathbb{Z} / (2) $ vector space.
6. We define addition in our $S(3,4,2t)$ Steiner system as follows: $0+v = v+0 = v$ for any $v$, $v+v = 0$ for any $v$, and, if $v$ and $w$ are different nonzero vectors, then $v+w$ is the third point of the Steiner system block determined by $v$ and $w$.
7. Commutativity of this addition law is easy to check. Every vector it its own additive inverse, so that's not an issue.
8. It remains to check associativity. It is easy to check that $ (a+b)+c = a+(b+c) $ when any one of $a$, $b$ or $c$ is zero. Therefore assume that they are all nonzero. It is likewise easy to prove associativity when any two of them are equal. Therefore assume they are distinct. It is easy to prove associativity when $a+b = c$ (that is, they form a block of the $S(2,3,2t-1)$ Steiner system). So assume that $a+b \neq c$.
9. How many fixed points does the pointwise stabilizer of $a$, $b$ and $c$ have? When $a$ and $b$ are fixed, $a+b$ also appears as a fixed point. Then, when $a$, $b$ and $c$ are fixed, we have $a$, $b$, $c$, $a+b$, $a+c$ and $b+c$ as fixed points. These must all be distinct (the cancellation laws are easy to verify for this addition, and the assumptions on $a$, $b$ and $c$ establish the rest of the distinctness), so we have at least 6 fixed points. However, $ (S_{2} \wr S_{t-2}) \cap A_{2t-4}$ is, when $t > 4$, non-regularly transitive on $2t-4$ points, so that a point stabilizer in it fixes only 2 points and the pointwise stabilizer of $a$, $b$ and $c$ (in the stabilizer of 0 in $K$) has 5 fixed points (and so is unable to accommodate our 6 fixed points). This is enough to contradict the existence of the $S(3,4,2t)$ Steiner system with these extra properties when $t > 4$, and so is enough to contradict the existence of $K$ when $t > 4$.

That gives a way of doing it by bare hands. Enjoy the tour of the small permutation groups you get out of this!