Separating closed $SO(p,q)$ orbits by invariant polynomials

Question

Consider the real Lie group $SO(p,q)$ (I believe that it happens to be a linearly reductive algebraic group over $\mathbb{R}$, if that's relevant). Also, if relevant, I'm mostly interested in the (Lorentzian) $p=1$ case. I'm also interested in the same question when $SO(p,q)$ is replaced by $O(p,q)$, but I suspect that the answers are similar.

Let $V$ be a finite dimensional representation of $SO(p,q)$. Is it true that the scalar $SO(p,q)$-invariant polynomials on $V$ separate the closed $SO(p,q)$-orbits on $V$?

I know that in general not all orbits (which are not necessarily closed) can be separated by polynomial invariants. But some results from geometric invariant theory do show that the closure of any orbit contains a unique closed orbit [Richardson & Solodowy (1990) JLMS 42 409-429]. So it's a given that any continuous (let alone polynomial) scalar invariant will not separate a non-closed orbit from the closed one in its closure. But there might still be hope to separate the closed orbits themselves.

What I already know is that the polynomial invariants do separate the closed orbits of the complexified representation $V\otimes \mathbb{C}$ of $SO(p,q;\mathbb{C}) \cong SO(p+q;\mathbb{C})$ and that the polynomial invariants of the $SO(p,q)$ representation are equivalent to the polynomial invariants of the $SO(p+q;\mathbb{C})$ representation. However, when we restrict to the smaller real group, the number of orbits may increase and the separation by polynomials is no longer obvious.

For instance, when $p=1$, the further restriction to the so called orthochronous subgroup $SO^{\uparrow}(1,q) \subset SO(1,q)$ does break polynomial separability already in the fundamental vector representation. In the language of special relativity, polynomial invariants can recognize whether vectors are timelike or spacelike, but cannot distinguish between future- and past-pointing timelike vectors. On the other hand, orthochronous transformations are precisely those that are not allowed to exchange future- and past-pointing timelike vectors. On the other hand, I do hope that problem does not occur for the slightly larger groups $SO(1,q)$ and $O(1,q)$.

Update: Friedrich Knop has answered the bulk of my question, based on the following observation:

But then there are also the real points of the algebraic group orbit $Gv$. They consist of all vectors of $V$ which can be obtained from $v$ by an element of the complex group $G(\mathbb{C})$. Hence $(Gv)(\mathbb{R})=G(\mathbb{C})v\cap V$. The latter are sometimes called the stable orbits.

The point is now that an invariant $f$ can only separate stable orbits. This is because $G(\mathbb{R})$-invariance implies $G$-invariance. Moreover, any two closed stable orbits can be separated by an invariant.

I'm now looking for a specific reference that discusses this fact and its proof.

Answer 1

The group $SO(p,q)$ is not per se an algebraic group. Rather there is an algebraic group $G$ such that $SO(p,q)=G(\mathbb R)$ is its group of real points. The main point is that also $G(\mathbb C)$ is defined which leads to two different concepts of orbits. First, there are the orbits $G(\mathbb R)v$ which you are probably after.

But then there are also the real points of the algebraic group orbit $Gv$. They consist of all vectors of $V$ which can be obtained from $v$ by an element of the complex group $G(\mathbb C)$. Hence $(Gv)(\mathbb R)=G(\mathbb C)v\cap V$. The latter are sometimes called the stable orbits.

The point is now that an invariant $f$ can only separate stable orbits. This is because $G(\mathbb R)$-invariance implies $G$-invariance. Moreover, any two closed stable orbits can be separated by an invariant.

The stable orbit is always a finite union of open $G(\mathbb R)$-orbits and these cannot be separated by invariants anymore. The good news is that an orbit $G(\mathbb R)v$ is (Hausdorff-)closed iff the associated stable orbit is (Hausdorff-)closed iff $Gv$ is (Zariski-)closed. This follows, e.g., form a theorem of Kempf on the existence of optimal $1$-parameter subgroups.

Now back to your real question: is it possible to make all closed $G(\mathbb R)$-orbits stable by enlarging the group, like from $SO^{\uparrow}(1,q)$ to $SO(1,q)$? This works in some cases but in general it is doomed to fail.

Example: Take the group $G(\mathbb R)=SO(1,q)$, $q\ge1$ acting on $\mathbb R^{1+q}=\mathbb R\mathbf e_0\oplus\mathbb R\mathbf e_1\oplus\ldots\oplus\mathbb R\mathbf e_q$. Put $V=S^2(\mathbb R^{1+q})$. The transformation $(x_0,x_1,x_2,\ldots,x_q)\mapsto(ix_1,ix_0,x_2,\ldots,x_q)$ is in $G(\mathbb C)$ and maps $v_1=\mathbf e_0^2$ to $v_2=-\mathbf e_1^2$. Thus $v_1$ and $v_2$ cannot be separated by an invariant but their $G(\mathbf R)$-orbits are closed and different (one is timelike the other spacelike).