I guess that the first motivation for geometric invariant theory was elementary geometry. From the point of view of the Erlangen program, a geometry is the study of the properties of a space that are invariant under a given group of transformations. Pretty close from the goal of geometric invariant theory, isn't it?

So for example, the only invariant of four points on the projective plane (under the projective group) is the cross-ratio (to be precise, any such rational invariant is an algebraic expression of the cross-ratio) . There are several invariants of two vectors in the euclidean plane: the inner product of the vectors, the (square of the norm of the) exterior product of these vectors and the ones obtained from the (square of the) norms of the two vectors, and of course any algebraic expression obtained from these invariants. There are none others and they give rise to the usual invariants of euclidean geometry, namely, length and angles.

Geometric invariant theory also classifies the relations between the invariants (and relations between the relations etc). For example, the ideal of relations between the invariants of two vectors in euclidean geometry is generated by the relation
$$
\parallel u \parallel^2 \parallel v \parallel^2 - \parallel u \times v\parallel^2 - \parallel u | v \ \parallel^2
$$
It can be shown that any theorem of projective plane geometry (either true or false) is equivalent to a relation between the invariants of the geometry. Here, the term "Theorem" should be understood in a geometric sense, as a (finite) succession of projective transformations that build points and lines from a given (finite) set of initial conditions.

For example the projective Pappus theorem is equivalent to the following (true) relation
$$
[ (b \wedge c') \wedge (b'\wedge c), (c \wedge a')\wedge (c' \wedge a), (a \wedge b')\wedge (a'\wedge b)] = [b',c',a][c',a',b][a',b',c][a,b,c] - [b,c,a'][c,a,b'][a,b,c'][a',b',c']
$$
where $a,b,c,a',b',c'$ are ( projective coordinates of the) points in the plane, $a\wedge b$ is (the coordinates in the dual of) the line determined by the points $a,b$, the wedge of two lines is the intersection point of the lines, and the bracket is the determinant of the three vectors representing the points in the projective plane. This may appear as a pretty complex relation between the coordinates of the points when expanded in polynomial form, but geometric invariant theory provides a neat classification of these relations.

good quotientof the semistable locus $X^{ss}$ exists. This idea was used by Mumford, Gieseker, etc to prove that a coarse moduli space $M_g$ exists. Nowadays, there are more fancy methods to prove this coarse moduli space exists, such as the Keel-Mori Theorem. $\endgroup$ – Ben Lim May 3 '15 at 16:59