$O(n)$ Polynomial invariant of symmetric tensors

Question

I apologize in advance if this question is not up to the level of research level questions on Math overflow. I am a complete outsider to invariant theory/representation theory and would like someone more knowledgeable than me to direct me to where I should read.

Let $Sym^k(\mathbb{R}^n)$ be the vector space of symmetric multilinear real valued maps in $k$ arguments. The orthogonal group, $O(n)$, acts on $\mathbb{R}^n$ and hence gives a natural action on $Sym^k(\mathbb{R}^n)$. Let $\mathcal{A}$ be the algebra of polynomials with domain $Sym^k(\mathbb{R}^n)$ and codomain $\mathbb{R}$. After a brief literature review, I believe Hilbert and others' work on classical invariant theory gives that the subalgebra of $\mathcal{A}^{O(n)}$ of polynomials that are constant on the orbits of action of $O(n)$ on $\mathcal{A}$ is finitely generated and "separates" the orbits.

Question: Is there an explicit finite list of generators of the algebra of $\mathcal{A}^{O(n)}$ ?

I seem to have an answer in case $k$ is even (assuming I didn't make any mistakes with identifications or other details):Set $k=2b$. A symmetric $\mathbb{R}$-multilinear map in $k$ arguments can (?) be naturally identified/thought of as a symmetric (?) endomorphism of $\otimes_{i=1}^{b}\mathbb{R}^n$, finally take your polynomials invariants to be the cofficents of the characteristic equation of that endomorphism on $\otimes_{i=1}^{b}\mathbb{R}^n$. My strategy is relying on the fact that any symmetric matrix with real entries can be diagonalized by an orthonormal eigenbasis

Question: Is the above argument correct for the case $k$ even, and what about the case $k$ is odd ? I admit I did not work out all details/identifications of the above argument yet so I imagine I could be wrong, but I will work them out asap.

Thank you,

Answer 1

Just in case this wasn't obvious from Abdemalek Abdesselam's answer, the following is a well-known fact. The algebra of invariant polynomials $\mathcal{A}^{O(n)}$ consists of linear combinations of all possible total contractions of any number of copies of the symmetric tensors from $Sym^k(\mathbb{R}^n)$. These contractions can be represented by $k$-valent graphs (draw a $k$-valent vertex for each copy of the symmetric tensor, with each attached leg representing an index, then connect all pairs of legs that correspond to an index contraction). Then, obviously, the algebra $Sym^k(\mathbb{R}^n)$ is generated by the connected graphs of this type (multiplication of invariants corresponds to the disjoint union of corresponding graphs). So you get an explicit lit of generators (the connected graphs), but this list is infinite. By the work of Hilbert, as you mention, there is a finite list of generators, but finding say a graphical representation of these generators is the difficult problem, with the answer (when known) being highly dependent on $n$ and $k$, as mentioned in the other answer.

The above well-known is a special case of the so-called First Fundamental Theorem of the invariant theory of $O(n)$. It can be challenging to find a clean statement of it of sufficient generality. We happen to have included such a statement in Appendix C of this paper with a self-contained proof. Actually, we stated it for the group $O(1,n-1)$, but the $O(n)$ case is completely analogous and even easier due to its compactness.

Answer 2

I am more familiar with $SL_n$ invariants of $n$-ary forms of degree $k$ rather than orthogonal invariants, so this is a very approximate answer. The dimension of the space of orbits is roughly $\binom{2b+n-1}{n-1}-\binom{n}{2}$ which is much bigger than the number $\binom{b+n-1}{n-1}$ of coefficients of the characteristic polynomial. So the answer to the second question is no. I think for the first question, generators are only known for low values of $n$ and $k$. You will find a wealth of information about orthogonal invariants on the page of one of the experts in the area, Marc Olive: http://w3.lmt.ens-cachan.fr/site/php_perso/perso_page_lmt.php?nom=OLIVE

For instance when $n=k=3$, the algebra of invariants has 13 generators given in terms of (finite-dimensional) Feynman diagrams in Theorem 4.2 of https://hal.archives-ouvertes.fr/hal-00827406/document