Continuous version of the fundamental theorem of invariant theory for the orthogonal group A standard result in the invariant theory of the orthogonal group states the following.
Theorem
Let $(E, \langle .,. \rangle)$ be an n-dimensional euclidean vector space,
let $f : E^m \rightarrow {\bf R}$
a polynomial function satisfying
$f(g(v_1), ... g(v_m)) = f(v_1,...,v_m)$
for all isometries $g$ of $E$ and $v_1$,..., $v_m \in E$.
Then such a function is a polynomial function
in the quantities $\{\langle{v_i}{v_j}\rangle\}_{i,j = 1...m}$.
Does the theorem holds in the topological setting,
namely when polynomial is replaced by continuous ?
My guess is that it should be true and the proof should be simpler than its algebraic counterpart, maybe a short computation using SVD.
All references I know present the algebraic proof though. Same question in the differential setting.
 A: Reflecting on the answer of Terence Tao, I guess it boils down to
the fact that an injective proper map between locally compact spaces
is a homeomorphism onto its image. Since we are working with
${\bf R}^n$ here, there is a simple characterisation of proper maps
that leads to the following statement.
Let $\Phi : {\bf R}^n \longrightarrow {\bf R}^k$ be a continuous
map satisfying
$$
\|\Phi(x)\| \longrightarrow \infty \quad when \quad  {\|x\| \rightarrow \infty}.
$$
Let us define the fiber relation on ${\bf R}^n$ by
$x \sim x' \iff \Phi(x) = \Phi(x').$
Then  $({\bf R}^n/\sim)$ is a locally compact metric space and
$\bar{\Phi} : ({\bf R}^n/\sim) \longrightarrow \Phi({\bf R}^n)$
is a homeomorphism.
The condition on the norm is there to ensure that
for all compact set $K \subset {\bf R}^k$, $\Phi^{-1}(K)$ is
closed and bounded (hence compact). In particular, the fibers
$\Phi^{-1}(\{y\})$ are compact and thus we can define
a distance on the quotient as follows:
$$
d(\bar{x}, \bar{x}') = d(\Phi^{-1}(\{\bar{\Phi}(\bar{x})\}), \Phi^{-1}(\{\bar{\Phi}(\bar{x}')\})).
$$
For the problem at hand, we take
$\Phi(v_1,...,v_l) = (\langle v_i, v_j \rangle)$
and note that the fibers of $\Phi$ are the orbits of the elements
of ${\bf R}^n$ under the action of the orthogonal group.
Then, for any invariant $f$, we have
$$
f(v_1,...,v_l) = \bar{f}(\bar{\Phi}^{-1}(\bar{\Phi}(\pi(v_1),..., \pi(v_l))))
               = \bar{f} \circ \bar{\Phi}^{-1}(\langle v_i, v_j\rangle).
$$
The norm condition also ensures that $\Phi({\bf R}^n)$ is closed,
so $\bar{f} \circ \bar{\Phi}^{-1}$ can be extended to all ${\bf R}^k$ if needed.
A: Yes.  It suffices to show that if one has a sequence $\vec v^{(n)} = (v^{(n)}_1,\dots,v^{(n)}_m) \in E^m$ whose Gram matrix $(\langle v^{(n)}_i, v^{(n)}_j \rangle)_{i,j=1,\dots,m}$ converges to a Gram matrix $(\langle v_i, v_j \rangle)_{i,j=1,\dots,m}$ of a tuple $\vec v = (v_1,\dots,v_m) \in E^m$, then after applying linear isometries to each of the $\vec v^{(n)}$, that $\vec v^{(n)}$ converges to $\vec v$.
By permuting indices we may assume that $v_1,\dots,v_k$ are linearly independent, and $v_{k+1},\dots,v_m$ are in the span of $v_1,\dots,v_k$, for some $0 \leq k \leq m$.  Applying the Gram-Schmidt process to $v_1,\dots,v_k$ and transforming $v^{(n)}_1,\dots,v^{(n)}_k$ appropriately we may assume that $v_1,\dots,v_k$ are orthonormal. It is not difficult to inductively apply isometries to the $\vec v^{(n)}$ so that $\vec v^{(n)}_i$ converges to $v_i$ for $i=1,\dots,k$.  If we write each $v_j, j=k+1,\dots,m$ as a linear combination $v_j = a_{j1} v_1 + \dots + a_{jk} v_k$ of $v_1,\dots,v_k$, then from the Gram matrix convergence we see that $\| v_j^{(n)} - (a_{j1} v^{(n)}_1 + \dots + a_{jk} v^{(n)}_k) \|^2$ converges to zero, hence $v_j^{(n)}$ converges to $v_j$ by the triangle inequality, and the claim follows.
