Maximal order of a metacyclic transitive permutation group of degree $n$

Question

What is the maximal order $f(n)$ of a metacyclic (metacyclic group is the extension of a cyclic group by a cyclic group) transitive permutation group of degree $n$? It can be easily proved that $f(p)=p(p-1)$ for prime number $p$, and the following argument (if it is correct) shows that $f(n)$ divides $n^2\phi(n)$:

For a metacyclic transitive permutation group $R$ of degree $n$, we can write $R=\langle a,b\rangle$ with $\langle a\rangle\trianglelefteq R$. Let $M=\langle a\rangle=Z_m$ and $L=\langle b\rangle=Z_l$. Since the point stabilizer of a transitive permutation group is core-free, we have $R_\omega\cap M=1$ for any point $\omega$. This implies that $M$ is semiregular and hence $$ m~\big|~n. $$ Since the center of $R$, denoted by $Z$, is semiregular, we have $|Z|~\big|~n$. Then $C\cap L\leqslant Z$ and $L/(C\cap L)\lesssim Aut(M)=Z_{\phi(m)}$ where $C$ is the centralizer of $M$ in $R$ yields $$ l~\big|~n\phi(m). $$ Therefore, $ml~\big|~n^2\phi(n)$ and thus $|R|~\big|~n^2\phi(n)$.

However, computation results suggest that $f(n)\leqslant n(n-1)$ for all $n$ and $f(n)=n(n-1)$ only if $n$ is prime.

Answer 1

I think I can prove that $f(n) \le n(n-1)$ with equality if and only if $n$ is prime, but it depends on the following result, for which I have not properly worked out the proof.

Let $H$ be the holomorph of the cyclic group $C_n$ of order $n$ - that is the semidirect product of $C_n$ with its automorphism group. Then the maximum order of elements in $H$ is $n$. I have checked by computer that it is true for $n \le 1200$, and I might try and work out the details later.

Anyway, let's assume that is true. If your normal subgroup $M = \langle a \rangle$ is transitive, then $M$ is self-centralizing in $S_n$, so we get $|R| \le n\phi(n)$.

Otherwise, suppose that $M$ has $n/m$ orbits of length $m$ with $m < n$. Then $b$ must act as a cycle of length $n/m$ on the orbits of $M$, so $c := b^{n/m}$ fixes each orbit of $M$. Furthermore, the actions of $c$ on these orbits are all equivalent, and so the the restriction of $c$ to each orbit has the same order. Since this restriction lies in the normalizer in $S_m$ of a cyclic subgroup of order $m$, it lies in the holomorph of $C_m$ and so by the (claimed) result above, this order is at most $m$. So the order of $b$ is at most $n$ and hence $|R| \le mn \le n^2/2$.

Note that, for $n = 2m$ with $m$ odd, there are examples of order $n^2/2$.

Answer 2

The following complements Derek's proof by showing the upper bound for element orders in the holomorph of $C_n$: Set $R=\mathbb Z/n\mathbb Z$. The holomorph of $C_n$ is $R\rtimes R^\times$. Pick $x=(a,b)\in R\rtimes R^\times$. Then $x^k=(a(1+b+\dots+b^{k-1}),b^k)$. Write $s_k=1+b+\dots+b^{k-1}$. Look at the $n$ terms $s_k$ for $k=1,\dots,n$. If it happens that two of them are equal, say $s_u=s_v$ for $1\le u\lt v\le n$, then $0=s_v-s_u=b^us_{v-u}$, and therefore $s_{v-u}=0$. If however these terms are pairwise distinct, then one of them must be $0$. So at any rate, there is $1\le k\le n$ with $s_k=0$. Furthermore, $0=(b-1)s_k=b^k-1$, so $x^k=1$.