Invariant theory over $\mathbb R$

Question

$\DeclareMathOperator\SO{SO}$Suppose we have a (continuous) linear action of $\SO(n,\mathbb R)$ on a vector space $\mathbb R^N$. Consider the ring of invariants $A\subset \mathbb R[x_1,\ldots, x_N]$, which is an $\mathbb R$-algebra. Is it true that the orbits of the $\SO(n,\mathbb R)$ action are in one-to-one correspondence with the $\mathbb R$-algebra homomorphisms $A\to\mathbb R$?

It is clear that for each orbit of the action we get such an evaluation homomorphism, by the value of an invariant polynomial on the orbit. But is the inverse true? Is there some nice book that considers that type of setting, over the real numbers?

Answer 1

The answer depends on what you mean by "one-to-one correspondence". Is it bijective or just injective? Robert Bryant's (standard) argument shows that $\mathbb R^N/\mathrm{SO}(n)\to \mathrm{AlgHom}_{\mathbb R}(A,\mathbb R)$ is injective. In general, this map is very far from being surjective, though. In other words, not every Homomorphism $A\to\mathbb R$ is an evaluation map. Already the standard action of $\mathrm{SO}(n)$ on $\mathbb R^n$ is a counterexample. In this case, $A=\mathbb R[q]$ where $q(v)=\|v\|^2$ is the norm square function. Thus a homomorphisms $f:A\to\mathbb R$ is given by the value $f(q)$ which can be an arbitrary real number. But the image of $\mathbb R^n/\mathrm{SO}(n)$ is obviously only $\mathbb R_{\ge0}$.

So what is the "correct quotient"? The answer is "It depends". A topologist would say it is the orbit space. The embedding into $\mathrm{AlgHom}_{\mathbb R}(A,\mathbb R)$ gives the orbit space the structure of a semialgebraic set on which you have notions of continuous or smooth function. A theorem of G. Schwarz states that all continuous/smooth invariants are pull-backs of continuous/smooth function on the orbit space.

An algebraist would prefer to call $\mathrm{AlgHom}_{\mathbb R}(A,\mathbb R)$ to be the quotient since it classified all closed orbits defined over $\mathbb R$ regardless whether the contain a real point or not.

A very nice paper calculating the image of $\mathbb R^N/\mathrm{SO}(n)\to \mathrm{AlgHom}_{\mathbb R}(A,\mathbb R)$ (in particular) in terms of inequalities is Procesi, Claudio; Schwarz, Gerald: Inequalities defining orbit spaces. Invent. Math. 81 (1985), 539–554

Answer 2

As YCor commented, the main point is to show that the invariant polynomials separate orbits. This follows from the compactness of $\mathrm{SO}(n)$. The point is this: Because $\mathrm{SO}(n)$ is compact, all of its orbits on $\mathbb{R}^N$ are closed. Hence if $B = \mathrm{SO}(n)\cdot b$ and $C = \mathrm{SO}(n)\cdot c$ are distinct orbits, there is a continuous function $f:\mathbb{R}^n\to \mathbb{R}$ such that $f(B)=0$ and $f(C)=1$. Also, because $B$ and $C$ are compact, the Weierstrass approximation theorem implies that $f$ can be approximated as closely as we like by a polynomial function on $\mathbb{R}^n$, say $g\in\mathbb{R}[x_1,\ldots,x_N]$ such that $g$ stays within $(-\epsilon,\epsilon)$ on $B$ and stays within $(1-\epsilon,1+\epsilon)$ on $C$, with $0<\epsilon< \frac12$. Now, using Haar measure on $\mathrm{SO}(n)$, take the average of $g$, yielding an $\mathrm{SO}(n)$-invariant polynomial $h\in \mathbb{R}[x_1,\ldots,x_N]$, such that $|h(B)|\le \epsilon$ and $|h(C)-1|<\epsilon$. Thus, $h$ separates the two orbits, as desired.