As Derek suggests in his comment, this question is too difficult to answer in general. However one could limit the question as follows: clearly if $K$ is a transitive permutation group then $|K|$ divides $|M|$ where $M$ is a maximal transitive subgroup of ${\mathrm Sym}(n)$; thus we can ask about the cardinality of a maximal transitive subgroup $M$ of ${\mathrm Sym}(n)$.
The O'Nan-Scott theorem is the main tool here. Roughly speaking it asserts that such a subgroup $M$ is either imprimitive (and hence a wreath product, with order formula easy), or else it is in a bunch of primitive families. Most of these families have a geometric description and, as such, it is easy to calculate their order.
The `difficult' family in this regard is the family of primitive almost simple groups. In this case one basically needs an enumeration of the maximal subgroups of all almost simple groups, which is a difficult problem but one which has received a great deal of attention.
Depending on what level of information you need, there are complete enumerations of maximal subgroups for many of the almost simples (although not all). However there are also some very nice general statements about the possible sizes of maximal subgroups. One example is this paper by Martin Liebeck; there are many others like it (many by Liebeck and his collaborators).
