I've often seen people in seminars justify the existence of a quotient of a scheme by an algebraic group by remarking that the group action is free. However, I'm pretty sure they are also invoking something else. So my question is: when you can you quotient a scheme by a free action and get a scheme? In particular, when do the coset spaces of a subgroup of an algebraic group exist as a scheme? And in these cases, how do you construct the quotient?

Hironaka's example of a proper nonprojective 3fold has a $\mathbb Z/2$ action which is free (actually, there's one fixed point, but you can just throw it out) whose quotient is not a scheme. (see an appendix of Hartshorne (page 443) or Shafarevich (page 75)) In general, the quotient of a scheme by a free action of a group is an algebraic space. If this group is finite, this is just because the action induces an etale equivalence relation (this is where you use freeness), and any quotient of a scheme by an etale equivalence relation is an algebraic space. For infinite groups, the quotient will be an algebraic stack fibered in sets, so it will be an algebraic space (c.f. this question). You are then reduced to the question, when is an algebraic space a scheme? Here's one situation where you get a scheme quotient. If you have a quasiprojective variety $X$ with an action of a connected reductive group $G$, then there is a geometric quotient $X/G$ if there exists a line bundle $L$ (with $G$action compatible with the $G$action on $X$) such that every point of $X$ is stable with respect to $L$. That is, if for every point $x\in X$, there is an invariant section $s$ of some tensor power of $L$ such that $X_s$ (the nonvanishing locus of $s$) is an open affine neighborhood of $x$ in which every $G$orbit is closed. This is Theorem 1.10 in Geometric Invariant Theory. I don't know if knowing that the action of $G$ is free helps in finding such an $L$. 


In the case of a quotient of a scheme $X$ by a finite group $G$, a necessary and sufficient condition for the quotient scheme $X/G$ to exist is that the orbit of every point of $X$ be contained in an affine open subset of $X$. This is proved in SGA I, Exposé V, proposition 1.8. In particular, if $X$ is a closed subscheme of projective space, then for any finite set of points of $X$ (in particular, any orbit of a single point) we can find a hyperplane not containing any of those points. The complement will be an affine open of $X$ containing the orbit. This covers the existence of the quotient in the cases that $X$ is projective (and by a similar argument, quasiprojective). Here is a sketch of the argument in SGA I: If the condition is satisfied, one first shows that this implies that $X$ is covered by $G$invariant open affines, then by taking invariants, constructs the quotient of each affine by $G$, and then glues together to get the global quotient $X/G$. Conversely, if $X/G$ is a scheme then for any open affine $V \subseteq X/G$ the inverse image of $V$ under the morphism $X\longrightarrow X/G$ is an open affine of $X$ stable under $G$. Since we can cover $X$ by such opens, the orbit of any point of $X$ is contained in some affine open set. As in George McNinch's answer there is no need for the $G$action to be free. (This discussion was limited to quotients by finite groups though!) 


There's an example of a free action with no geometric quotient here, in example 18. 


To answer (in one case) the particular question it is always true for linear algebraic groups that the orbit space of a closed subgroup is a scheme, and it will also be a linear algebraic group provided that the subgroup is normal. I am not sure about the first part... If the action is transitive often one can argue that the quotient is representable by an algebraic space and then use transitivity of the group action to show that one in fact has a scheme. I am pretty sure freeness should correspond to good properties of the geometric quotient provided one exists (probably only then on some open subscheme) rather than guaranteeing existence and it seems that this is the case given the example Charles linked to. 


I think that the quotient of a quasiprojective scheme by a finite action is always a scheme. I don't know a reference for this. 


Well, this question is old, but I suppose something got updated so I noticed it. There are various statements made in answers/comments about finite groups, and I wanted to mention that Serre's text "Algebraic Groups and Class Fields" contains a reference. In it, Serre shows (in III.3.12) that:
Serre is working with algebraic varieties over an algebraically closed field $k$  i.e. reduced, separated $k$schemes V of finite type. In this setting there is no need to stipulate that the action of the finite group G on V be free. 


A standard case where the quotient exists is when the group G is finite. In this case, if you start with the ring of functions O, and take the ring of G invariants O^G. Then O^G is finitelygenerated and is Spec(O^G) is the quotient of Spec(O). Its worth noting that these quotients behave a little different that you might expect. For instance, the quotient can still be very nice when there are stablizers. Take Z/2 acting on C[x] by x > x. Then the ring of invariants is C[x^2], and the quotient map is the map from affine space to itself which takes a point p to p^2. In the algebraic case, the fixed point at the origin isn't 'orbifoldy' enough to be a singularity. For nonfinite actions, this invariant based approach can still be rigged to work, but its trickier. This typically goes under the name Geometric Invariant Theory, and the main concern is finding the right locus of bad points to discard before quotienting. 

