Closed orbits of complete flags in $\mathbb{C}^n$

Let $B$ be a symmetric (or antisymmetric) non-degenerate bilinear form on $\mathbb{C}^n$ and let $G$ be the associated group of automorphisms $O(n)$ (resp. $Sp(n)$). What can we say about the Zariski-closed $G$-orbits on the variety of complete flags in $\mathbb{C}^n$ for these two cases?

I believe the whole flag variety is an $O(n)$-orbit for the symmetric case, but I'm not sure about the antisymmetric case. My apologies if this is basic; I posted this over at math.se earlier this month, but received no replies.

The whole flag variety is definitely not a single $O(n)$-orbit. You can see this already in the case of $n=2$, where the flag variety is just $\mathbb P^1$, and there are two orbits: $\{\pm i\}$ and everything else. More generally, by Witt's theorem, the $O(n)$-orbit on the grassmannian of $r$-planes through any given totally isotropic subspace of dimension $r$ is the (closed!) subvariety of all totally isotropic subspaces of dimension $r$. So you will have 'the' variety of isotropic flags as a closed $O(n)$-orbit on the flag variety. What's going on here is that a closed $O(n)$-orbit in the flag variety will be projective, hence will look like $O(n)/P$ for some parabolic, i.e. it will be a partial flag variety for $O(n)$. The same kind of thing will be true for $Sp(n)$, with Lagrangian subspaces taking the place of isotropic subspaces.

Anyway, there is a unified framework for dealing with these types of questions. The key observation is that $O(n)$ and $Sp(n)$ (if $n$ is even) are symmetric subgroups of $GL(n)$, i.e., each can be realized as the set of fixed points of an algebraic involution $\theta \colon GL(n) \to GL(n)$, namely $\theta(g)=g^{-t}$ and $\theta(g)=Jg^{-t}J^{-1}$, resp., where $J$ is the matrix of the symplectic form defining $Sp(n)$. Your question is thus a special case of the following:

If $K=G^\theta$ is a symmetric subgroup of $G$, what can we say about the $K$-orbits on $G/B$?

The answer is: lots! A good starting point is the set of notes to Allen Knutson's course on the topic. Be sure to also take a look at the Richardson--Springer paper linked to on that page.

• In the case of P1 the action is transitive, no? Mar 24, 2014 at 4:03
• @Mariano: No, it's not: if $g$ is in $O(2)$ then $g \cdot i = (\det g) i = \pm i$. Mar 24, 2014 at 4:09
• There seems to be a misunderstanding here. Do you actually mean $O(n)$? How does it act? Should it not be $U(n)$ instead? Mar 24, 2014 at 6:15
• @AlexDegtyarev: I think that the question is being asked in the algebraic category, so that $G$ is a complex algebraic group. The notation $O(n)$ is unclear, but means $O(n,\mathbb{C})$. Mar 24, 2014 at 7:51
• Yes, $O(n)$ here is really $O(\mathbb C^n, B)$, as stated in the OP. Mar 24, 2014 at 13:56

$O(n)$, by definition preserves the bilinear forms. Consider a subspace $W_r$ of dimension $r$ where the form is zero for any pair of vectors. Put it as the $r$th term of a complete flag $F$. Now consider another complete flag where for the $r$th term $W_r'$ the bilinear form is not identically zero. A form-preserving automorphism of the full vector space cannot take $W_r$ to $W_r'$, so there is no transitive action.

In Bull American Math Soc, you will find a paper of Joseph Wolf which provides an answer to your question. best jorge

• Would you mind adding a more precise reference and possibly a link? I could not find the paper based on the information you gave... Apr 1, 2014 at 20:49