I had been looking lately at Sylow subgroups of some specific groups and it got me to wondering about why Sylow subgroups exist. I'm very familiar with the proof of the theorems (something that everyone learns at the beginning of their abstract algebra course)  incidentally my favorite proof is the one by Wielandt  but the statement of the three Sylow theorems still seems somewhat miraculous. What got Sylow to imagine that they were true (especially the first  the existence of a sylow subgroup)? Even the simpler case of Cauchy's theorem about the existence of an element of order $p$ in a finite subgroup whose order is a multiple of $p$ although easy to prove (with the right trick) also seems a bit amazing. I believe that sometimes the hardest part of a proving a theorem is believing that it might be true. So what can buttress the belief for the existence of Sylow subgroups?

1$\begingroup$ Have you ever seen the proof of Sylow in the excercises to Jacobson's Basic Algebra I? It's one of the slickest proofs I've ever seen (I mean, I did the problem, but he leads you to this slick solution.) $\endgroup$– Harry GindiMar 19 '10 at 5:24

1$\begingroup$ @fpqc  I think you're talking about Gallagher's proof of a generalization: the number of subgroups of order $p^k$, where $p^k$ divides $G$, is 1 mod p. $\endgroup$– Steve DMar 19 '10 at 5:51

6$\begingroup$ Qiaochu: That the existence of Sylow subgroups is true for abelian group doesn't strike me as a good reason to expect it to be true in general finite groups. In a finite abelian group there is a subgroup of every size which divides the size of the group. That's certainly not true in finite groups in general. And I don't think Sylow could have been inspired by any analogy with eigenspace decompositions; the question asked what Sylow's motivation was. $\endgroup$– KConradMar 19 '10 at 18:59

2$\begingroup$ I just looked at the very nice paper by Waterhouse "The Early Proofs of Sylow's Theorem" which, among other things gives Sylow's original proof which is quite nice, and in fact corresponds to the way that modern computer algebra packages find Sylow Subgroups. It also refers to a (by now obscure) result of Cauchy, where, among other things he gives an explicit construction for the pSylow subgroups of $S_n$. $\endgroup$– Victor MillerMar 19 '10 at 19:33

2$\begingroup$ This question would benefit from a title in the form of a question $\endgroup$– j.c.Mar 26 '10 at 13:36
Victor, you should check out Sylow's paper. It's in Math. Annalen 5 (1872), 584594. I am looking at it as I write this. He states Cauchy's theorem in the first sentence and then says "This important theorem is contained in another more general theorem: if the order is divisible by a prime power then the group contains a subgroup of that size." (In particular, notice Sylow's literal first theorem is more general than the traditional formulation.) Thus he was perhaps in part inspired by knowledge of Cauchy's theorem.
Sylow also includes in his paper a theorem of Mathieu on transitive groups acting on sets of primepower order (see p. 590), which is given a new proof by the work in this paper. Theorems like Mathieu's may have led him to investigate subgroups of primepower order in a general finite group (of substitutions).

$\begingroup$ Thanks, I'm looking at it now. I also found a paper by Scharlau "The Discovery of the Sylow Theorems" (in German). The first paragraph of the Math. Reviews review is: The author evokes primarily the works of Sylow that antedate 1872—the date of the first publication of Sylow’s famous theorems —and proposes to provide answers to the following questions: How did Sylow arrive at the formulation of his theorems, what were the mathematical techniques at his disposal, what was the influence of his contemporaries, and what hopes did Sylow have for his work? $\endgroup$ Mar 19 '10 at 21:06

1$\begingroup$ What I like about Sylow's paper is that he uses Cauchy's theorem of the existence of an element of order $p$ to "boot strap" up to a maximal $p$ subgroup. Once one accepts Cauchy's result it makes clear why you would expect such subgroups to exist. Since Cauchy proved it by giving an explicit construction for $S_n$ Frobenius' proof is interesting since it gives a way of transferring that construction to a general finite group. $\endgroup$ Mar 19 '10 at 21:34
The Sylow theorems are finite group analogues of a bunch of results about "maximal unipotent subgroups" in algebraic groups. Basically, the Sylow subgroups play a role analogous to the role played by the maximal unipotent subgroups.
In the case where the group is the general linear group, the maximal unipotent subgroup can be taken as the group of upper triangular matrices with 1s on the diagonal, for instance. There are existence, conjugacy, and domination results for these analogous to the existence, conjugacy, and domination part of Sylow's theorems: maximal unipotents exist, every unipotent is contained in a maximal unipotent, all maximal unipotents are conjugate. The role analogous to "order" is now played by "dimension".
The normalizer of the Sylow subgroup plays the role of the maximal connected solvable subgroup, also called the Borel subgroup (see Borel fixedpoint theorem and LieKolchin theorem). In the case of the general linear group, this is the group of upper triangular invertible matrices.
There are similar results for Lie algebras too, basically arising from Engel's theorem and Lie's theorem.
In fact, much of the study of simple groups and their geometry relies on this geometric interpretation of Sylow subgroups, psubgroups, and their normalizers. This deeper study of the geometry/combinatorics of simple groups is called local analysis in group theory and is closely related to the recently popular topic of "fusion systems" which are essentially studying the conjugation action of a group on subgroups of a particular Sylow subgroup.
ADDED BASED ON COMMENT BELOW: For a finite field $F_q$ where q is a power of p, the maximal unipotent subgroup of $GL_n(F_q)$ is the $p$Sylow subgroup. I had originally intended to mention this, but forgot.

$\begingroup$ All this, though, cannot have been what suggested the Sylow theorem to Sylow! $\endgroup$ Mar 26 '10 at 2:02

$\begingroup$ Yes, that's true, because Sylow was thinking of things in terms of permutation groups, and to the best of my knowledge did not explore the connection with linear groups or algebraic groups. Still, it's worth noticing that there are multiple reasons why one might suspect a statement to be true, or at least plausible. $\endgroup$ Mar 26 '10 at 2:08

1$\begingroup$ That's a very good point! By the way, something that you probably thought too obvious to mention when revealing this remarkable analogy, is that for GL_n(F_p), maximal unipotent subgroups ARE Sylow subgroups :) $\endgroup$ Mar 26 '10 at 11:25

$\begingroup$ Yes, I had forgotten to mention that. Not just GL_n(F_p), but also GL_n(F_q) where q is a power of p. $\endgroup$ Mar 26 '10 at 13:30
An extension of the Vipul's ideas can be found in the article (couldn't find a link to the pdf with google)
Subgroup complexes by Peter Webb, pp. 349365 in: ed. P. Fong, The Arcata Conference on Representations of Finite Groups, AMS Proceedings of Symposia in Pure Mathematics 47 (1987).
But as Mariano already commented, the analogy to the maximal unipotent subgroups of the general linear group was probably not Sylow's motivation. As commented before, he was maybe looking for maximal $p$subgroups (i.e., maximal with respect to be a $p$subgroup).
This is also the leitmotif of my favorite proof of the Sylow theorems given by Michael Aschbacher in his book Finite Group Theory. It is based on Cauchy’s theorem (best proved using J. H. McKay’s trick to let $Z_p$ act on the set of all $(x_1, \dots, x_p) \in G^p$ whose product is $1$ by rotating the entries) and goes essentially like this:
The group $G$ acts on the set $\mathrm{Syl}_p(G)$ of its maximal $p$subgroups by conjugation. Let $\Omega$ be a (nontrivial) orbit with $S\in\Omega$. If $P$ is a fixed point of the action restricted to $S$ then $S$ normalizes $P$ and $PS=SP$ is a $p$group. Hence $P=S$ by maximality of both $P$ and $S$, and $S$ has a unique fixed point. As $S$ is a $p$group, all its orbits have order $1$ or a multiple of $p$, in particular $\mathrm{Syl}_p(G) = 1 \bmod p$. All orbits of $G$ are disjoint unions of orbits of $S$ proving $\Omega = 1 \bmod p$ and $\Omega' = 0 \bmod p$ for all other orbits $\Omega'$ of $G$. This implies that $\Omega = \mathrm{Syl}_p(G)$, as $\Omega$ was an arbitrary nontrivial orbit of $G$, showing that the action of $G$ is transitive. The stabilizer of $S$ in $G$ is its normalizer $N_G(S)$, and as the action is transitive $G:N_G(S) = \mathrm{Syl}_p(G) = 1 \bmod p$. It remains to show that $p$ does not divide $N_G(S):S=N_G(S)/S$. Otherwise, by Cauchy’s theorem there exists a nontrivial $p$subgroup of $N_G(S)/S$ whose preimage under the projection $N_G(S) \to N_G(S)/S$ is $p$subgroup properly containing $S$ contradicting the maximality of $S$.
I don't know if this was the original motivation, but this has some interesting motivating ideas: Abstract nonsense versions of "combinatorial" group theory questions