Let $G$ be a transitive permutation group on a finite set $\Omega$. It is clear that if $G$ is regular, then every proper subgroup of $G$ is intransitive. Is there any other class of groups with this property? I mean that which transitive groups has no proper transitive subgroup?

4$\begingroup$ This is just the definition of a minimal transitive group. It probably would be better to just ask for a characterization of minimal transitive subgroups of ${\rm Sym}(\Omega)$. This will depend on arithmetical properties of $\Omega$. So, for example, if $\Omega = p^{n}$ for some prime $p$ and integer $n$, then a minimal transitive group on $\Omega$ will be a $p$group. $\endgroup$ – Geoff Robinson Apr 14 '16 at 18:53

$\begingroup$ Dear Geoff, You are right. Thank you for your comment. $\endgroup$ – majid arezoomand Apr 15 '16 at 2:50

$\begingroup$ Question: what's the smallest possible value of $\Omega$? The smallest prime power seems to be 32. $\endgroup$ – YCor Apr 15 '16 at 18:28

1$\begingroup$ @YCor, if you mean what is the smallest degree of a transitive group with no regular subgroup, it is 6. There are five examples of degree $6$, including for example $A_6$ and (the smallest, by order) $A_4$ on 6 points. The smallest example of primepower degree has order $32$ and degree $8$. $\endgroup$ – verret Apr 15 '16 at 19:26
Consider the symmetric group $S$ on $n$ symbols in its action on the $k$subsets of $\{1,\ldots,n\}$. If $k\ne2,4$ and $n=2k+1$, then the only transitive subgroups of $S$ are $S$ itself and the alternating group. Hence in this case the alternating group acts transitively but all its proper subgroups are intransitive. See my "More Odd Graph Theory", Discrete Math. 32 (1980) 205207. The basic idea is that a group that is transitive on $k$subsets must be $(k1)$transitive and in most cases this leaves with only the symmetric and alternating groups.

$\begingroup$ Dear Chris, Thank you for your answer. $\endgroup$ – majid arezoomand Apr 15 '16 at 2:56
As Geoff Robinson has already written in his comment, you are asking for minimally transitive permutation groups. Specifically you ask whether there is any class of such groups other than the regular permutation groups. The answer to this is a clear yes.  For example there are up to conjugacy only $51$ regular permutation groups of degree $32$, but there is a total of $11605$ conjugacy classes of minimally transitive permutation groups of degree $32$. See Page 3 of the paper The Transitive Permutation Groups of Degree 32 by Cannon and Holt which reports about the determination of all $2801324$ conjugacy classes of transitive permutation groups of degree $32$.

$\begingroup$ Dear Stefan, Thank you for your answer $\endgroup$ – majid arezoomand Apr 15 '16 at 2:57
Whenever one has a vertextransitive graph which is not a Cayley graph, one has an example of such a group (maybe more than one). E.g. both $A_5$ and $5:4$ act vertextransitively on the Petersen graph, and are minimal transitive subgroups of $S_{10}$. See e.g. the paper by B.D.McKay and C.E.Praeger.

$\begingroup$ Dear Dima, Thank you for your answer. $\endgroup$ – majid arezoomand Apr 15 '16 at 2:58
As was implicit in my comment every transitive permutation group contains such a subgroup. I single out the case that $\Omega$ has prime power cardinality in my comment ( in which case any minimal transitive subgroup of ${\rm Sym}(\Omega)$ is a $p$group, using Sylow's theorem). Just one further remark: in the case $\Omega = p^{n}$ for a prime $p$ and positive integer $n$, a transitive $p$subgroup $G$ of ${\rm Sym}(\Omega)$ is a minimal transitive subgroup if and only if each pointstabilizer $G_{\omega}$ is contained in the Frattini subgroup $\Phi(G)$.
For if $G$ has a transitive proper subgroup $H$, then we have $HG_{\omega} = G$, so certainly $G_{\omega} \not \leq \Phi(G)$ as $G = \langle H, G_{\omega} \rangle \neq H$.
On the other hand, if $G$ is minimal transitive, and $M$ is a maximal subgroup of $G$, then $MG_{\omega} \neq G$ (otherwise $M$ would be transitive), so that $MG_{\omega} = M$ by maximality ( for $M \lhd G$, so that $MG_{\omega}$ is a subgroup of $G$). Hence $G_{\omega} \leq M$, so that $G_{\omega} \leq \Phi(G),$ as $M$ was an arbitrary maximal subgroup of $G$.
Later note: notice then that when $G$ is a finite $p$group, there is a bijection between faithful transitive permutation representations of $G$ in which no proper subgroup is transitive, and $G$conjugacy classes of subgroups $X \leq \Phi(G)$ such that $X \cap Z(G) = 1.$
For if $X$ is such a subgroup, then ${\rm core}_{G}(X) = 1$ ( otherwise, the core meets $Z(G)$ nontrivially), so the (transitive) permutation action of $G$ on the (say right) cosets of $X$ in $G$ is faithful. Each point stabilizer in this action is a $G$conjugate of $X$, so is contained in $\Phi(G) \lhd G$ as $X \lhd G$. Hence by the above remark, $G$ is a minimal transitive permutation group on the cosets of $X$.
On the other hand, if $G$ has a faithful minimal transitive action on the right cosets of its subgroup $Y$, then $Y \leq \Phi(G)$ as $Y$ is a point stabilizer. Also $Y$ is corefree in $G$ since the permutation action is faithful, so we certainly have $Y \cap Z(G) = 1$.

$\begingroup$ Dear Prof. Robinson, Thank you very much for your exact answer. $\endgroup$ – majid arezoomand Apr 16 '16 at 10:52

$\begingroup$ @ GeoffRobinson : Your proof works for the the more general case of nilpotent subgroups of $S_{p^n}$. I mean that we can say that a transitive nilpotent subgroup of $S_{p^n}$ is a minimal transitive subgroup if and only if each pointstabilizer contained in the Frattini subgroup. $\endgroup$ – majid arezoomand Apr 16 '16 at 11:21

$\begingroup$ But it was already noted in my comment that a minimal transitive subgroup of $S_{p^{n}}$ is necessarily a $p$group anyway. $\endgroup$ – Geoff Robinson Apr 16 '16 at 11:35

$\begingroup$ Yes, you are right. I should edit my comment. I think that I should replace $S_{p^n}$ with $S_\Omega$, where $\Omega$ is arbitrary. $\endgroup$ – majid arezoomand Apr 16 '16 at 11:57
Here's a description in the case when $\Omega$ is a prime power. As mentioned by Geoff Robinson, this case boils down to the case when we have an acting $p$group.
Fix a prime $p$. Let $G$ be a $p$group and $A=G/\Phi$ is its maximal $p$elementary abelian quotient (so $\Phi$ is the Frattini subgroup), and $H$ a subgroup of $G$.
That $G$ is faithful on $G/H$ means that $H\cap Z(G)=1$ (standard).
That every proper subgroup is intransitive means that $H$ is contained in the Frattini subgroup $\Phi$.
Proof: (This is already in Geoff Robinson's post.) Indeed, if $H$ is not contained in $\Phi$, let $B$ be its projection on $A$, and write $A=B\oplus C$, and let $L$ be the inverse image of $C$ in $G$; this is a normal subgroup of $G$, so $LH$ is a subgroup of $G$. Since the projection of $LH$ on $A$ is all of $A$, we have $LH=G$. Conversely, if $H$ is contained in $\Phi$ and $M\subset G$ is transitive on $G/H$, then $MH=G$, which, in projection to $A$, implies that the projection of $M$ on $A$ is equal to $A$, which in turn implies that $M=A$. $\Box$
To get examples, we need groups $G$ for which the Frattini subgroup $\Phi$ containains elements of order $p$ that are not central in $G$. This is clearly impossible if $\Phi$ is central in $G$, which excludes examples of order $p^k$ for $k\le 3$. For $k=4$, it gives examples for $p\ge 3$: denoting by $C_p$ the cyclic group the wreath product $C_p\wr C_p$ has order $p^4$, its center is cyclic of order $p$ while its Frattini subgroup is isomorphic to $C_p^2$. On the other hand, it yields no example of order 16, that is, in every group of order 16, every element of order 2 in the Frattini subgroup is central (I did a casebycase check but without a computer verification, please tell me if you check it's correct or not). Anyway there are examples of order 32, as already mentioned (for instance, the semidirect product $C_2^3\rtimes C_4$ where $C_4$ acts by a unipotent Jordan matrix; in this case, $H$ can be chosen to be $(\{0\}\times C_2\times \{0\})\rtimes C_2$ which defines an action of a set of cardinal 8).

2$\begingroup$ I said some (but not all) of these things in my answer (not comment) below. $\endgroup$ – Geoff Robinson Apr 15 '16 at 17:37

A very repeatable construction to yield such permutation groups is to let a group act on the cosets of a small subgroup.
The idea is that if $H$ is a subgroup of $G$, a subgroup $T \leq G$ acts transitively on the cosets of $H$ if and only if $HT = G$. If $H$ is small, $T$ is very restricted (beginning, for example, with $[G:T] \leq H$) and it's easy, in many cases, to prove $T$ must be the whole $G$. If $G$ is regular, this corresponds to choosing $H = 1$, which forces $T=G$ without further discussion.