Invariants of symmetric forms with respect to the symplectic group Take a 6-dimensional vector space $V$ (for simplicity, over $\mathbb{C}$) and play the following game (for example, by employing the online Lie program): consider the 21-dimensional space $S^2V^*$ of symmetric two-forms on $V$ and decompose the space $S^k(S^2V)$ of degree-$k$ homogeneous polynomials on $S^2V^*$ into irreducible $\mathsf{SL}_6$-modules and, simultaneously, into irreducible $\mathsf{Sp}_6$-module, with $k=1,2,3,4,5,6$. The number of one-dimensional constituents you'll obtain is the following:


*

*For $\mathsf{SL}_6$ there is a unique one-dimensional constituent $\langle d\rangle$, that appears when $k=6$;

*For $\mathsf{Sp}_6$ the first one-dimensional constituent $\langle p\rangle$ pops up with $k=2$, then a second one $\langle q\rangle$ with $k=4$, accompained by $\langle p^2\rangle$, and, finally, for $k=6$, there are three one-dimensional constituents: $\langle p^3\rangle$, $\langle p q\rangle$ and $\langle d\rangle$.


Now it is well known that $d$ is the determinant.

QUESTION: what about the $\mathsf{Sp}_6$-invariants $p$ and $q$ of a symmetric two-form $\alpha$ on $V$? Can we read them off from the characteristic polynomial of a suitable endomorphism of $V$ related to $\alpha$? Does anybody know where precisely in the literature this is discussed? (Should be classical.)

In particular, I'm interested in the normal forms of elements $\alpha\in S^2V^*$ with rispect to the symplectic group: in the case of the linear group, the normal form of $\alpha$ is simply a diagonal matrix with as many 1's on the diagonal as the rank of $\alpha$ - but if the group is smaller I expect a more involved outcome.
 A: Nice question!
More generally, let $V=\mathbb{C}^{2n}$. Consider the $2n\times 2n$ matrix
$$
\varepsilon=\begin{pmatrix}
0 & I_n \\
-I_n & 0
\end{pmatrix}
$$
and the symplectic group ${\mathsf{S}\mathsf{p}}_{2n}$ which preserves the fundamental alternating bilinear form with matrix $\varepsilon$. An element $F$ of the symmetric power $S^p(V^{\vee})$ can be seen as a homogeneous polynomial $F(x)$ of degree $p$ in the variable $x=(x_1,\ldots,x_{2n})$.
It also corresponds to a unique symmetric array
$$
(F_{i_1,\ldots,i_p})_{(i_1,\ldots,i_p)\in [2n]^p}
$$
where $[2n]$ denotes the set of allowed index values $\{1,2,\ldots,2n\}$.
Symmetric means the entries stay the same if one permutes the $p$ indices.
The correspondence is so that the identity
$$
F(x)= F_{i_1,\ldots,i_p} x_{i_1}\cdots x_{i_p}
$$
holds. Note that I used Einstein's convention where indices $i_1,\ldots,i_p$ are to be summed independently over the set $[2n]$. I will keep using this convention below.
Now for integers $q,r,\ell$ with $0\le \ell\le\min(q,r)$, one can define a "symplectic transvectant" which is a ${\mathsf{S}\mathsf{p}}_{2n}$-equivariant map
$S^q(V^{\vee})\times S^r(V^{\vee})\rightarrow S^{q+r-2\ell}(V^{\vee})$. To a pair of forms $F$, $G$, we associate the new form
$$
H(x)= F_{i_1,\ldots,i_q} G_{j_1,\ldots,j_r} \varepsilon_{i_1,j_1}\cdots
\varepsilon_{i_{\ell},j_{\ell}}\ x_{i_{\ell+1}}\cdots x_{i_q}\ 
x_{j_{\ell+1}}\cdots x_{j_r}
$$
I will write $(F,G)_{\ell}$ for this new form $H$.
Now suppose $p$ is even. Then for any $m\ge \frac{p}{2}$, one has a linear endomorphism
$$
\begin{array}{cccc}
\mathcal{L}_{n}^{F}: & S^{m}(V^{\vee}) & \longrightarrow & S^{m}(V^{\vee}) \\
 \ & G & \longmapsto & (F,G)_{\frac{p}{2}}
\end{array}
$$
which depends on the choice of $F$.
Let $\mathscr{H}_{m,s}(F)$ denote the coefficient of $\lambda^s$ in essentially the characteristic polynomial ${\rm det}(Id-\lambda \mathcal{L}_{n}^{F})$.
Alternatively, let $\mathscr{P}_{m,s}(F)$ denote the trace of the $s$-th power
of $\mathcal{L}_{n}^{F}$. It is not hard to see that $\mathscr{H}_{m,s}(F)$
and $\mathscr{P}_{m,s}(F)$ are ${\mathsf{S}\mathsf{p}}_{2n}$-invariants of $F$. They give you
one-dimensional submodules in $S^{s}(S^{p}(V))$.
The above is a trivial generalization to the symplectic context of a construction in the invariant theory of binary forms (the ${\mathsf{S}\mathsf{p}}_{2}={\mathsf{S}\mathsf{L}}_{2}$ case) due to Hilbert in his  Königsberg Habilitationsschrift. I studied these concrete invariants in my recent article
"An algebraic independence result related to  a conjecture of Dixmier on binary form invariants" in Res. Math. Sci. 2019.
The preprint version is here.
The main result I proved in that article is that for $n=1$, and for $p=2k$ with $k$ even, the invariants $\mathscr{P}_{k,2},\mathscr{P}_{k,3},\ldots,\mathscr{P}_{k,k+1}$ are algebraically independent. Note that this trivially shows the same holds true for any $n\ge 1$, by specializing to a generic form $F$ which only depends on the variables $x_1,x_{n+1}$.
Note that one can also represent the invariants graphically, as in the picture

which is taken from the above article. In the left picture, the lines with arrows correspond to $\varepsilon$'s, and the boxes correspond to symmetrizations.
Now take $n=3$, $p=2$, $m=\frac{p}{2}=1$, which gives $\mathscr{P}_{1,s}(F)={\rm tr}((\varepsilon F)^s)$, where $F$ is viewed as a $6\times 6$ symmetric matrix. These are the invariants you see in the Lie program calculations. Clearly, they vanish unless $s\ge 2$ is even.
For $p=2$, general $n$. The first fundamental theorem (FFT) of invariant theory for ${\mathsf{S}\mathsf{p}}_{2n}$ easily implies that the particular invariants $\mathscr{P}_{1,s}$, $s\ge 1$ generate the ring of invariants. Because of the relations between power sum symmetric functions, and the remark about parity, one has for this ring the list of generators
$$
\mathscr{P}_{1,2},\mathscr{P}_{1,4},\mathscr{P}_{1,6},\ldots,\mathscr{P}_{1,2n}.
$$
They are algebraically independent. Indeed, take $F$ to be the quadratic form with matrix
$$
\begin{pmatrix}
0 & D \\
D & 0
\end{pmatrix}
$$
where $D$ is the diagonal matrix with entries $y_1,\ldots,y_n$. Then the above invariants specialize to the power sums in the variables $y_1^2,\ldots,y_n^2$.
So this gives a complete description of the ring of invariants.
For a quick sketch of a proof of the FFT for ${\mathsf{S}\mathsf{p}}_{2n}$
see:
Invariants for the exceptional complex simple Lie algebra $F_4$
It proceeds by reduction to the FFT for ${\mathsf{S}\mathsf{L}}$ and/or ${\mathsf{G}\mathsf{L}}$ which are proved in
How to constructively/combinatorially prove Schur-Weyl duality?
and
How to constructively/combinatorially prove Schur-Weyl duality?
