Consider the real Lie group $SO(p,q)$ (I believe that it happens to be a linearly reductive algebraic group over $\mathbb{R}$, if that's relevant). Also, if relevant, I'm mostly interested in the (Lorentzian) $p=1$ case. I'm also interested in the same question when $SO(p,q)$ is replaced by $O(p,q)$, but I suspect that the answers are similar.

Let $V$ be a finite dimensional representation of $SO(p,q)$. Is it true that the scalar $SO(p,q)$-invariant polynomials on $V$ separate the closed $SO(p,q)$-orbits on $V$?

I know that in general not all orbits (which are not necessarily closed) can be separated by polynomial invariants. But some results from geometric invariant theory do show that the closure of any orbit contains a unique closed orbit [Richardson & Solodowy (1990) JLMS 42 409-429]. So it's a given that any continuous (let alone polynomial) scalar invariant will not separate a non-closed orbit from the closed one in its closure. But there might still be hope to separate the closed orbits themselves.

What I already know is that the polynomial invariants do separate the closed orbits of the complexified representation $V\otimes \mathbb{C}$ of $SO(p,q;\mathbb{C}) \cong SO(p+q;\mathbb{C})$ and that the polynomial invariants of the $SO(p,q)$ representation are equivalent to the polynomial invariants of the $SO(p+q;\mathbb{C})$ representation. However, when we restrict to the smaller real group, the number of orbits may increase and the separation by polynomials is no longer obvious.

For instance, when $p=1$, the further restriction to the so called orthochronous subgroup $SO^{\uparrow}(1,q) \subset SO(1,q)$ does break polynomial separability already in the fundamental vector representation. In the language of special relativity, polynomial invariants can recognize whether vectors are timelike or spacelike, but cannot distinguish between future- and past-pointing timelike vectors. On the other hand, orthochronous transformations are precisely those that are not allowed to exchange future- and past-pointing timelike vectors. On the other hand, I do hope that problem does not occur for the slightly larger groups $SO(1,q)$ and $O(1,q)$.

Update: Friedrich Knop has answered the bulk of my question, based on the following observation:

But then there are also the real points of the algebraic group orbit $Gv$. They consist of all vectors of $V$ which can be obtained from $v$ by an element of the complex group $G(\mathbb{C})$. Hence $(Gv)(\mathbb{R})=G(\mathbb{C})v\cap V$. The latter are sometimes called the stable orbits.

The point is now that an invariant $f$ can only separate stable orbits. This is because $G(\mathbb{R})$-invariance implies $G$-invariance. Moreover, any two closed stable orbits can be separated by an invariant.

I'm now looking for a specific reference that discusses this fact and its proof.