Since the comments are already getting long, I'll add this in community-wiki format to clarify a few points. I should emphasize that I'm not at all a specialist in GIT but have dealt with neighborhing problems involving algebraic groups.

First, there are fundamental differences between characteristic 0 (where Mumford mostly worked) and prime characteristic. In the former case, a linear algebraic group splits as a semidirect product of its unipotent radical and a reductive (Levi) subgroup which is unique up to conjugacy. In the latter case, such a splitting may fail. More important here is the fact that "reductive" is equivalent in characteristic 0 to "linearly reductive": all finite dimensional representations as an algebraic group are completely reducible. This fails badly in prime characteristic.

On the other hand, unipotent groups have the nice property that their algebraic actions have only closed orbits. (And their only irreducible representation is the trivial one.)

In terms of classical invariant theory, all reductive groups (even in prime characteristic) turn out to have the good property that the associated rings of polynomial invariants are finitely generated. (This was settled in prime characteristic by Haboush's proof using algebfaic geometry of the Mumford Conjecture, that reductive implies "geometrically reductive". There is an algebraic proof in Jantzen's book Representations of Algebraic Groups.)

Anyway, Mumford's initial slim volume has grown in its third edition to a larger book, with contributions first by Fogarty and then by Kirwan. But the prefaces Mumford wrote show pretty clearly what he had in mind and why he wanted to work especially with reductive groups. While unipotent groups have their own interest, much classical work on moduli problems involves reductive groups. Their actions involve orbits which need not be closed, as already seen in the adjoint representation (where only the semisimple elements live in closed orbits). So Mumford's ideas about stable and semistable points are delicate, but important to sort out.