Let $\mathfrak{g}$ be the Lie algebra of $GL(n,\mathbb{R})$. Let $\theta(X) =  X^T$ be the Cartan involution on $\mathfrak{g}$; it induces decomposition as $\mathfrak{g} = \mathfrak{k} \oplus \mathfrak{p}$ of $+1$eigenspace $\mathfrak{k}$ and $1$ eigenspace $\mathfrak{p}$. Then $\mathfrak{k}$ is the Lie algebra of $K = O(n)$. I want to compute the dimensions of $K$invariants $\textrm{Hom}_K(\wedge^q \mathfrak{p}, \mathbb{C})$, which, I suppose, is equal to $(W/ \mathfrak{k} W)^*$ where $W = \wedge^q \mathfrak{p}$, where $\mathfrak{p}$ is viewed as $\mathfrak{k}$module by adjointrepresentation. Could you someone point a way further?

$\begingroup$ when you said "compute the invariants" what exactly are you looking for? $\endgroup$ – Abdelmalek Abdesselam Dec 14 '17 at 14:19

$\begingroup$ I see that you updated your question to mean the dimension of the space of invariants. In that case, I think my answer completely solves that. The dimension is the number of integer partitions of $q$ with distinct parts that are all congruent to 1 mod 4. More precisely, this is the case if $n\ge q$. Let me know if you have any questions about my argument. $\endgroup$ – Abdelmalek Abdesselam Dec 14 '17 at 19:52
To offer a slightly more geometric viewpoint on the same, the space $\bigoplus_q \mathop{\mathrm{Hom}}_K(\Lambda^q(\mathfrak{p}),\mathbb{C})$, which is the direct sum of all spaces you are considering, is the cohomology of the Lie algebra $\mathfrak{gl}_n$ relative to its subalgebra $\mathfrak{o}_n$, which, by a standard argument, is the same as $H^*(U(n)/O(n))$. It is well known (it really is; if you need some reference, see, for example, page 28 of lecture notes of Haefliger  http://www1.mat.uniroma1.it/people/piazza/HaeDC.pdf) to be the exterior algebra generated by elements $\xi_{4k3}$, for all $k$ such that $2k1\le n$. In fact, $\xi_{d}$ is the antisymmetrisation of the trace of product of $d$ matrices. This confirms the very nice answer given by Abdelmalek Abdesselam.

$\begingroup$ Thanks for bringing more perspective to my bare hands brute force approach... I knew from looking at the yellow book by Loday a long time ago that Lie algebra cohomology is related to invariants of matrices as studied by Procesi, Rasmyslov, Formanek and others but I didn't study that connection in earnest. $\endgroup$ – Abdelmalek Abdesselam Dec 15 '17 at 18:11

1$\begingroup$ @Dotsenko : I was looking at Vogan's article where this isomorphism is sketched. Your reference helped a lot. Thank you. $\endgroup$ – user49908 Dec 16 '17 at 4:37

$\begingroup$ @AbdelmalekAbdesselam : to be fair, a lot of computations of Lie algebra cohomology need results like Procesi/Rasmyslov/Formanek if one needs to explain them to "hardcore algebraists". However, in many cases using algebraic topology gives alternative proofs (e.g. via the Serre spectral sequence for various fibrations). $\endgroup$ – Vladimir Dotsenko Dec 16 '17 at 5:14
You are basically looking for trivial $O(\mathbb{C}^n)$ representations contained in $\Lambda^k \mathfrak{p}^*.$ Since your representation $\mathfrak{p}$ is the space of symmetric matrices, you can find the whole decomposition into irreducibles of $\Lambda^k S^2 \mathbb{C}^n$ as a special case of plethysm for the Lie group $O(\mathbb{C}^n).$ In fact, this very special case of plethysm was already dealt with on this site for the group $GL(\mathbb{C}^n)$ in (at least) two questions:
Known decomposition of $\bigwedge^k Sym^d \mathbb C^n$ in special cases?
What is known about this plethysm?
Please note, that irreducible $GL(\mathbb{C}^n)$ representations decompose further under restriction to $O(\mathbb{C}^n)$  see e.g. Symmetry, Representations, and Invariants by Nolan Wallach and Roe Goodman.
To check whether you get correct decompositions and/or multiplicities you can use mathematical software. See e.g.:
Symmetric powers of Schur polynomials
I've whipped up some Sage code which however seems to be quite slow: https://cocalc.com/share/25e624ba9091484faf2e71deb7120f58/trivial%20reps%20in%20exterior%20algebra.sagews?viewer=share

$\begingroup$ seems I misread the question the matrices are indeed symmetric instead of antisymmetric. $\endgroup$ – Abdelmalek Abdesselam Dec 14 '17 at 13:06


$\begingroup$ Thank you I was thinking along similar lines but not beyond the following: By schur's lemma it following that the given object is equal to the $\mathbb{C}$ times multiplicity of trivial representation in $\wedge^k \mathfrak{p}$. Then I thought, "IF" adjoint representation of $\mathbb{k}$ in $\mathfrak{p}$ is irreducible then we get zero for $k =1 $ and by duality for $k= n1$. I could not proceed further. Thanks for the insight. $\endgroup$ – user49908 Dec 14 '17 at 14:12

$\begingroup$ @user49908 The representation of $O(V)$ on symmetric matrices is not irreducible. The symmetric form defining $O(V)$ is invariant! $\endgroup$ – Vít Tuček Dec 14 '17 at 15:57

1$\begingroup$ Pattern $1,0,0,0,1$ for $q=1,2,3,4,5$ is confirmed on these few cases. However, for $B_3$ I got "Multiplicity of the trivial representation in the 14th exterior power: 2" $\endgroup$ – Vít Tuček Dec 14 '17 at 16:20
Edit: I misread the question and understood $\mathfrak{p}$ as corresponding to antisymetric instead of symmetric matrices. So what immediately follows answers a different question. See the end of this post for a complete solution of the OP's case.
I don't know what you mean exactly by "compute the invariants": perhaps finding an explicit linear basis? Essentially $\wedge^q\mathfrak{p}$ is the same as the antisymmetric plethysm $\wedge^q(\wedge^2(\mathbb{R}^n))$. If I remember correctly, this kind of objects appears in relation to the Hodge conjecture for certain Abelian varieties as in work by Ken Ribet and Fumio Hazama. Namely, in some particular situations, the invariant elements in such a plethysm are the Hodge classes in the cohomology of the variety. See in particular the article "Branching rules and Hodge cycles on certain Abelian varieties" in AJM 1988. There could be some useful tools for your question in that literature.
As for a linearly generating set of invariants, it is easy to construct as follows. Let me call a Wick contraction any set partition $P$ of $\{1,2,\ldots, 2q\}$ with blocks of size two. Let me denote by $P_{\ast}$ the special case $$ \{\{1,2\},\{3,4\},\ldots,\{2q1,2q\}\}\ . $$ For indices $i,j$ in $\{1,\ldots,n\}$, let me use the Kronecker delta notation $\delta_{i,j}$ for the indicator function of the condition $i=j$. For a Wick contraction $P$ and a collection of $2q$ indices $i_1,\ldots,i_{2q}$ in $\{1,\ldots,n\}$, let me write $$ A(P)_{i_1,i_2,\ldots,i_{2q}}=\prod_{\{\alpha,\beta\}\in P} \delta_{i_{\alpha},i_{\beta}}\ . $$ One can view $\wedge^q(\wedge^2(\mathbb{R}^n))$ as the real vector space of all arrays $T=(T_{i_1,\ldots,i_{2q}})_{i_1,\ldots,i_{2q}\in\{1,\ldots,n\}}$ which change sign if one exchanges indices within a block of $P_{\ast}$ or if one rigidly exchanges two blocks. Now define the linear forms $L_P$ indexed by Wick contractions given by $$ L_P(T)=\sum_{i_1,\ldots,i_{2q}\in\{1,\ldots,n\}} A(P)_{i_1,i_2,\ldots,i_{2q}} \ T_{i_1,i_2,\ldots,i_{2q}}\ . $$ By the first fundamental theorem of 19th century invariant theory for $K=O(n)$, these form a linearly generating collection for ${\rm Hom}_K(\wedge^q\mathfrak{p},\mathbb{C})$. Of course one can restrict to $P$'s which are "transverse" to $P_{\ast}$, i.e. , such that $P\cap P_{\ast}=\varnothing$. I don't immediately see how to prune the remaining collection to get a basis, since I expect tons of linear relations. The stable case $n\ge 2q$ might be easier to analyze first.
In the OP's case the relevant space is $\Gamma:=\wedge^q({\rm Sym}^2(\mathbb{R}^n))$. The description of a generating set of invariants is exactly the same. The only thing that changes is the identification of $\Gamma$ with a space of arrays $T$. In this situation $T_{i_1,\ldots,i_{2q}}$ must stay the same if two indices in the same block of $P_{\ast}$ are exchanged and it must change sign if two blocks are exchanged. Also the transversality condition can be dropped.
Note that one can reduce the generating set a bit as follows. Instead of Wick contractions (aka perfect matchings) $P$ one can index invariants by integer partitions $\lambda$ of $q$. If one takes $T=A^{(1)}\wedge\cdots\wedge A^{(q)}$ for a bunch of symmetric matrices $A^{(1)},\ldots,A^{(q)}$ then the invariant encoded by $\lambda$ is the complete antisymmetrization of a "manual polarization" of $$ \prod_{a}\ {\rm tr}(A^{\lambda_a})\ . $$ By "manual polarization" I mean replace each $A$ factor by an $A^{(k)}$ so everyone is used exactly once. If one thinks of $P$'s as fixedpointfree involutions, take the cycle type of the composition $P\circ P_{\ast}$ and divide all the cycle lengths by two (extra care needed if lack of transversality). That gives you the partition $\lambda$. It is easy to see that the invariant $L_{\lambda}$ vanishes unless all the parts $\lambda_a$ are congruent to 1 mod 4. It also vanishes if there are repeated (odd) parts. Indeed if one does a cyclic permutation along a trace of length $\lambda_a$ one gets the identity $L_{\lambda}=(1)^{\lambda_a1}L_{\lambda}$. So an even part kills the invariant. For $\lambda_a$ congruent to 3 mod 4 one can do a "mirror symmetry" of the cycle which fixes a vertex. This involves $(\lambda_a1)/2$ disjoint transpositions which is an odd number. If there are two cycles of length $\lambda_a$ then exchanging them gives $(1)^{\lambda_a^2}=1$ since we reduced the discussion to odd parts.
So your space of invariants $(\wedge^q({\rm Sym}^2(\mathbb{R}^n)))^{O(n)}$ is of dimension 1 for $q=1$. Then it vanishes for $q=2,3,4$. I suspect it has dimension 1 for $q=5$ and $n$ not too small. By the above it has at most that dimension. Showing equality means showing that the antisymmetrization of
$$
{\rm tr}(A^{(1)}A^{(2)}A^{(3)}A^{(4)}A^{(5)})
$$
is not identically zero.
Let $N(q)$ be the number of integer partitions $\lambda$ of $q$ with distinct parts that are congruent to 1 mod 4. The argument I just gave proves the upper bound $$ {\rm dim}(\wedge^q({\rm Sym}^2(\mathbb{R}^n))^{O(n)})\le N(q)\ . $$ Interestingly, the computations by Vit also give a computer assisted proof that, in fact, equality holds for $q\le 14$ (and $n$ large enough, i.e., in the stable case).
Update: In the trace of five matrices case one can prove nonvanishing as follows. The invariant $L_\lambda$ corresponding to $\lambda=(5)$ is $$ L_{\lambda}(T)=\frac{1}{5!}\sum_{\sigma\in\mathfrak{S}_5} \varepsilon(\sigma)\ {\rm tr}(A^{(\sigma(1))}\cdots A^{(\sigma(5))}) $$ when $T=A^{(1)}\wedge\cdots\wedge A^{(5)}$. Now take the following $5\times 5$ matrices. Let $A^{(1)}_{12}=A^{(1)}_{21}=1$ and all other entries zero. Let $A^{(2)}_{23}=A^{(2)}_{32}=1$ and all other entries zero. Etc. Let $A^{(5)}_{51}=A^{(5)}_{15}=1$ and all other entries zero. Namely, consider the graph given by the cycle $(12345)$. For each edge in the cycle take the adjacency matrix of the corresponding oneedge graph. If I did not mess up my computation, the corresponding specialization of $L_{\lambda}$ is equal to $1/12$.
This can be generalized for a partition $\mu$ of $q$, also with distinct parts congruent to 1 mod 4. One then defines an element $T_{\mu}=A^{(1)}\wedge\cdots\wedge A^{(q)}$ as before. Namely, build a graph $G$ with vertex set $\{1,\ldots,q\}$ made of disjoint cycles with lengths given by the parts of $\mu$. For each edge in $G$ build the adjacency matrix of the corresponding oneedge graph to get the $A$ matrices. If one has a part $\mu_a=1$ corresponding to a cycle $(j)$ then take the matrix with $A_{jj}=1$ and all other entries zero. A quick combinatorial argument shows that $L_{\lambda}(T_{\mu})$ is an invertible diagonal matrix (with rows and columns indexed by partitions). By construction products of $A$'s give zero most of the time unless $\lambda$ replicates $\mu$. This immediately implies that $N(q)$ is the exact answer to the OP's question.