I've often seen people in seminars justify the existence of a quotient of a scheme by an algebraic group by remarking that the group action is free. However, I'm pretty sure they are also invoking something else. So my question is: when you can you quotient a scheme by a free action and get a scheme? In particular, when do the coset spaces of a subgroup of an algebraic group exist as a scheme? And in these cases, how do you construct the quotient?
7 Answers
In the case of a quotient of a scheme $X$ by a finite group $G$, a necessary and sufficient condition for the quotient scheme $X/G$ to exist is that the orbit of every point of $X$ be contained in an affine open subset of $X$. This is proved in SGA I, Exposé V, proposition 1.8.
In particular, if $X$ is a closed subscheme of projective space, then for any finite set of points of $X$ (in particular, any orbit of a single point) we can find a hyperplane not containing any of those points. The complement will be an affine open of $X$ containing the orbit. This covers the existence of the quotient in the cases that $X$ is projective (and by a similar argument, quasiprojective).
Here is a sketch of the argument in SGA I: If the condition is satisfied, one first shows that this implies that $X$ is covered by $G$invariant open affines, then by taking invariants, constructs the quotient of each affine by $G$, and then glues together to get the global quotient $X/G$.
Conversely, if $X/G$ is a scheme then for any open affine $V \subseteq X/G$ the inverse image of $V$ under the morphism $X\longrightarrow X/G$ is an open affine of $X$ stable under $G$. Since we can cover $X$ by such opens, the orbit of any point of $X$ is contained in some affine open set.
As in George McNinch's answer there is no need for the $G$action to be free. (This discussion was limited to quotients by finite groups though!)
Hironaka's example of a proper nonprojective 3fold has a $\mathbb Z/2$ action which is free (actually, there's one fixed point, but you can just throw it out) whose quotient is not a scheme. (see an appendix of Hartshorne (page 443) or Shafarevich (page 75))
In general, the quotient of a scheme by a free action of a group is an algebraic space. If this group is finite, this is just because the action induces an etale equivalence relation (this is where you use freeness), and any quotient of a scheme by an etale equivalence relation is an algebraic space. For infinite groups, the quotient will be an algebraic stack fibered in sets, so it will be an algebraic space (c.f. this question).
You are then reduced to the question, when is an algebraic space a scheme?
Here's one situation where you get a scheme quotient. If you have a quasiprojective variety $X$ with an action of a connected reductive group $G$, then there is a geometric quotient $X/G$ if there exists a line bundle $L$ (with $G$action compatible with the $G$action on $X$) such that every point of $X$ is stable with respect to $L$. That is, if for every point $x\in X$, there is an invariant section $s$ of some tensor power of $L$ such that $X_s$ (the nonvanishing locus of $s$) is an open affine neighborhood of $x$ in which every $G$orbit is closed. This is Theorem 1.10 in Geometric Invariant Theory. I don't know if knowing that the action of $G$ is free helps in finding such an $L$.
Well, this question is old, but I suppose something got updated so I noticed it. There are various statements made in answers/comments about finite groups, and I wanted to mention that Serre's text "Algebraic Groups and Class Fields" contains a reference. In it, Serre shows (in III.3.12) that:
 the quotient of an affine variety by a finite group is again an affine variety. (This is basically Greg Muller's answer).
 if V is an algebraic variety on which the finite group G acts, and if each Gorbit is contained in an affine open subvariety of V, then the quotient of V by G is an algebraic variety. (In Hironaka's example, mentioned by Anton Geraschenko, there are orbits not contained in any affine open  see Remark 2 on p. 80 in the reference [Shafarevich, Basic Algebraic Geometry 2] given in Anton Geraschenko's answer).
 in particular, if V is a quasiprojective variety (locally closed in some projective space $\mathbf{P}^r$), and if a finite group $G$ acts on $V$, then every $G$orbit is contained in an affine open subvariety and hence (as in David Speyer's answer) the quotient of V by the action of G exists.
Serre is working with algebraic varieties over an algebraically closed field $k$  i.e. reduced, separated $k$schemes V of finite type.
In this setting there is no need to stipulate that the action of the finite group G on V be free.
To answer (in one case) the particular question it is always true for linear algebraic groups that the orbit space of a closed subgroup is a scheme, and it will also be a linear algebraic group provided that the subgroup is normal.
Suppose that we have a closed subgroup H of G. One way to build the orbit space is as a Gorbit in the projective bundle P(V) associated to some rational representation G > GL(V). The point is that we can pick such a representation having the property that V has a one dimensional subspace W so that H is precisely the subgroup of G fixing W. Taking [W] in the associated projective space of V we get that H is the isotropy group of [W]. One can then identify G/H with the orbit of [W] in P(V) and it can be checked that this is the right thing to do in terms of the universal property. In particular the orbit space is quasiprojective and one can go onto study when the orbit space is projective (i.e. look at parabolic subgroups).
I am not sure about the first part... If the action is transitive often one can argue that the quotient is representable by an algebraic space and then use transitivity of the group action to show that one in fact has a scheme. I am pretty sure freeness should correspond to good properties of the geometric quotient provided one exists (probably only then on some open subscheme) rather than guaranteeing existence and it seems that this is the case given the example Charles linked to.
There's an example of a free action of a (nonreductive) subgroup $\left( \begin{array}{cc}1&*\\0&1\end{array} \right) \subset \operatorname{SL}_2$ on an affine variety $A$ with no geometric quotient, see Derksen Quotients of algebraic group actions, example 18. The variety $A$ consists of pairs of matrices $3 \times 6$ and $6 \times 3$, whose product is a certain explicitly given matrix $3 \times 3$.
I think that the quotient of a quasiprojective scheme by a finite action is always a scheme. I don't know a reference for this.

$\begingroup$ In the projective case, I think the argument goes like this. Consider the Hilbert scheme of G points in your projective scheme X. The action of G on X induces an action on this Hilbert scheme, and the closed subscheme fixed by G is naturally X/G because a point of the Hilbert scheme which is fixed by G is the same as a Ginvariant set of G points (i.e. a Gorbit). I don't see how to extend this to the quasiprojective case because I don't see why you should be able to find a compactification where you can extend the Gaction so that it's still free. $\endgroup$ Oct 22, 2009 at 6:11

1$\begingroup$ @Anton, in the quasiprojective case, you can use the same argument. For any quasiprojective variety, there is a quasiprojective Hilbert scheme of $\lvert G \rvert$ points on $X$. For example, you can take the Hilbert scheme of points on the projective closure $\overline X$ and remove the closed subscheme of points which intersect $\overline X  X$. Hilbert schemes of higherdimensional varieties don't work well in quasiprojective varieties, but for points, there's no problem. $\endgroup$ Dec 8, 2011 at 20:27
A standard case where the quotient exists is when the group G is finite. In this case, if you start with the ring of functions O, and take the ring of G invariants O^G. Then O^G is finitelygenerated and is Spec(O^G) is the quotient of Spec(O).
Its worth noting that these quotients behave a little different that you might expect. For instance, the quotient can still be very nice when there are stablizers. Take Z/2 acting on C[x] by x > x. Then the ring of invariants is C[x^2], and the quotient map is the map from affine space to itself which takes a point p to p^2. In the algebraic case, the fixed point at the origin isn't 'orbifoldy' enough to be a singularity.
For nonfinite actions, this invariant based approach can still be rigged to work, but its trickier. This typically goes under the name Geometric Invariant Theory, and the main concern is finding the right locus of bad points to discard before quotienting.

3$\begingroup$ Even if the group is finite, there may not be a quotient in the category of schemes. The procedure you describe produces a quotient of an affine scheme by a finite group, but strange things can happen if you act on nonaffine schemes, like the example of Hironaka I mentioned in my answer. $\endgroup$ Oct 21, 2009 at 4:11