To supplement the answer given by Francois, I'd emphasize that the question is essentially algebraic (over an algebraically closed field of any characteristic) in the spirit of the Borel-Chevalley structure theory. In this general setting, the older Borel notes and his second edition book, along with my book and Springer's, provide similar treatments of Borel subgroups or larger parabolic subgroups in a reductive algebraic group. Along the way it turns out that only the quotients $G/H$ with $H$ parabolic can be projective varieties (so this condition characterizes parabolics). In particular, the kind of embedding in projective space described in the question only allows an orbit to be closed (hence projective) if the isotropy group is parabolic. (In my book, this is developed in 21.3.)

Aside from that, the study of orbits and invariants in such representation-theoretic situations has been actively pursued in a lot of papers, mostly concentrating on the classical characteristic 0 setting (where for instance reductive groups behave better). Some of the influential work has been done by Hanspeter Kraft, Claudio Procesi, Gerald Schwarz, along with the Russian school: Ernest Vinberg, Vladimir Popov, Dmitri Panyushev, etc.