Is it possible to classify finite simple groups whose every maximal subgroups are not of prime order? Is it possible to answer to this question in the class of finite groups?
Prop: If $G$ is a finite simple group, then a maximal subgroup of $G$ is trivial or has composite order
Proof: A maximal subgroup of $G$ being trivial clearly corresponds to $G$ being cyclic of prime order. Assume, then, that $G$ is nonabelian.
If $G$ has a maximal subgroup $C$ of prime order, then the action of $G$ on cosets of $C$ is Frobenius. Thus $G$ is a Frobenius group, $C$ is a Frobenius complement and $G$ contains a Frobenius kernel, i.e. $G$ is not simple. QED
So this answers your first question. As for your more general question about finite groups. Well, again, if a group has a maximal subgroup of prime order, then it is Frobenius, so you should consult the literature on Frobenius groups. For this I particularly recommend Isaac's "Finite Group Theory" and Passman's "Permutation groups".
Examples of groups with a maximal subgroup of prime order include dihedral groups of order $2m$ ($m$ odd) or, more generally $C_n \rtimes C_p$ where $p$ is a prime and $C_p$ acts semiregularly on $C_n$.

$\begingroup$ There can be maximal subgroups of prime order which are normal. $\endgroup$ – Geoff Robinson Jan 3 '13 at 21:35

$\begingroup$ You do need to consider the (easy) case of a maximal subgroup of prime order which is normal. Given that, you have basically proved then that a finite group which has a maximal subgroup of prime order either has order $pq$ for (not necessarily distinct) primes $p$ and $q,$ or else is a Frobenius group with complement of prime order $p$ acting irreducibly on an elementary Abelian Frobenius kernel. For in the Frobenius group case, the complement is only maximal if the kernel is a $q$group for some prime $q,$ by Theorems about coprime operator groups, and you reach the conclusion I stated. $\endgroup$ – Geoff Robinson Jul 5 '14 at 18:36

$\begingroup$ To continue the above comment, for example, the only dihedral groups which have a maximal subgroup of prime order have order $2p$ for some prime (allow $p=2$ if you consider a Klein $4$group as dihedral). $\endgroup$ – Geoff Robinson Jul 6 '14 at 5:59
Late answer, in a somewhat different direction from Nick Gill's answer: (Note, by the way, that an Abelian finite simple group has its only maximal subgroup (the trivial group), not of prime order).
Using some more sophisticated group theory, it is possible to prove (and is reasonably wellknown to specialists) that if $M$ is a nilpotent maximal subgroup of a finite group $G$ with $F(G) = 1$ and $G = [G,G]$, then $M$ is a nonAbelian $2$group of order at least $16.$
For note that if $P \in {\rm Syl}_{p}(M)$ for an odd prime $p,$ and $P \neq 1,$ then $M = N_{G}(P)$ since $N_{G}(P) < G.$ Hence $P \in {\rm Syl}_{p}(G).$ Then $M = N_{G}(ZJ(P)),$ and $M$ certainly has a normal $p$complement. Hence $G$ has a normal $p$complement by a Theorem of Thompson. This is a contradiction. Hence $M$ is a $2$group (and is a Sylow $2$subgroup of $G$).
If $M$ is Abelian, then $G$ has a normal $2$complement by similar reasoning. If $M$ is quaternion of order $8,$ then we see that $M = N_{G}(U)$ whenever $1 \neq U \leq M.$ By Alperin's Fusion Theorem, $G$ has a normal $2$complement.
If $M$ is dihedral of order $8,$ then Alperin's fusion theorem again leads to a contradiction ( see, eg, Gorenstein's book on Finite Groups), for $\{U,V,M \}$ is a conjugation family for $M,$ where $U$ and $V$ are the two Klein 4subgroups of $M.$ Since $M = N_{G}(U) = N_{G}(V)$ by maximality of $M,$ we see that $G$ has a normal $2$complement, a contradiction. Hence $M$ is a nonAbelian $2$group of order at least $16,$ as claimed.
Finally, it is well known that the finite simple group ${\rm PSL}(2,17)$ has a maximal subgroup which is dihedral of order $16,$ so the occurrence of $16$ in the statement is not arbitrary.