reductive group orbits in P(V)?

Question

Say $G$ is a reductive group over $\mathbb{C}$. We can take a dominant highest weight $\lambda$ and look at the action of $G$ on $X = \mathbb{P} V(\lambda)$. The stabilizer of the class of the highest weight vector is a parabolic subgroup so the orbit is isomorphic to $G/P$. What about the other orbits in $X$? If $[v] \in X - G/P$ then is there a good description of $G.[v]$ or its closure? If this is hard for general $[v] \in X - G/P$ are there any conditions you can put on $[v]$ that make it easier? Is $G/P$ the only closed orbit?

Does anyone know of references that address these questions?

Answer 1

Yes, the projectivized orbit of a highest weight vector is well known to be the only closed one. It's also an orbit under the compact real form $G_u$, and as such is the unique $G_u$-orbit that is a complex (hence Kähler) submanifold (work of Borel, Weil, Tits, Hirzebruch). In this paper Kostant and Sternberg work out the conditions under which $G_u\cdot[v]$ is a symplectic submanifold. For an exposition see e.g. Guillemin and Sternberg's Symplectic Techniques in Physics.

(Orbits under other real forms are thoroughly studied in J. A. Wolf, The action of a real semisimple group on a complex flag manifold.)

Your problem, however, is that of studying the orbits in the flag manifold $\Bbb P(V)=GL(V)/GL(V)_{[w]}$ under any complex (irreducibly represented, reductive) $G\hookrightarrow GL(V)$. In this generality I'm not sure there is much one can say. For example, a theorem of Chevalley (J. Humphreys, Linear algebraic groups, 11.2) asserts that every $G/H$, $H$ Zariski closed in $G$, occurs as such an orbit for some (not necessarily irreducible) representation.

Answer 2

To supplement the answer given by Francois, I'd emphasize that the question is essentially algebraic (over an algebraically closed field of any characteristic) in the spirit of the Borel-Chevalley structure theory. In this general setting, the older Borel notes and his second edition book, along with my book and Springer's, provide similar treatments of Borel subgroups or larger parabolic subgroups in a reductive algebraic group. Along the way it turns out that only the quotients $G/H$ with $H$ parabolic can be projective varieties (so this condition characterizes parabolics). In particular, the kind of embedding in projective space described in the question only allows an orbit to be closed (hence projective) if the isotropy group is parabolic. (In my book, this is developed in 21.3.)

Aside from that, the study of orbits and invariants in such representation-theoretic situations has been actively pursued in a lot of papers, mostly concentrating on the classical characteristic 0 setting (where for instance reductive groups behave better). Some of the influential work has been done by Hanspeter Kraft, Claudio Procesi, Gerald Schwarz, along with the Russian school: Ernest Vinberg, Vladimir Popov, Dmitri Panyushev, etc.