Invariant theory deals with an algebraic, geometric or analytic structure X , submited to the action of an (algebraic) group G . It studies G-invariant elements of X as well as the set of G-orbits.
Invariant theory exists since the 19th century (Symbolic calculus of invariants). It deals with an algebraic, geometric or analytic structure X , submited to the action of an (algebraic) group G . Its first purpose is the study of G-invariant elements of X. For instance, are they finitely generated ? ("Hilbert's 14th problem",see V.L. Popov, Soviet Math.-Dokl.20, 1979)). What is the structure of the set X/G of orbits ? In case X is an algebraic variety, is X/G an algebraic variety ? (cf."Geometric Invariant Theory", David Mumford and al., Springer 1994). Other references are : The Classical Groups (H. Weyl, Princeton Univ. Press 1946,1997); Invariant Theory, old and new (J.A. Dieudonné, J.B. Carrell, Acad.Press 1971); Invariant theory (V.L. Popov, E.B. Vinberg,in Alg.Geom.IV, Enc.Math.Sci.,vol.55, Springer 1994);Lectures on Invariant Theory ( I. Dolgachev, Cambr.Univ.Press, 2003.).