Return to Answer

Edit: I misread the question and understood $\mathfrak{p}$ as corresponding to antisymetric instead of symmetric matrices. So what immediately follows answers a different question. See the end of this answer for a complete solution of the OP's case.

As for a linearly generating set of invariants, it is easy to construct as follows. Let me call a Wick contraction any set partition $P$ of $\{1,2,\ldots, 2q\}$ with blocks of size two. Let me denote by $P_{\ast}$ the special case $$ \{\{1,2\},\{3,4\},\ldots,\{2q-1,2q\}\}\ . $$ For indices $i,j$ in $\{1,\ldots,n\}$ let me use the Kronecker delta notation $\delta_{i,j}$ for the indicator function of the condition $i=j$. For a Wick contraction $P$ and a collection of $2q$ indices $i_1,\ldots,i_{2q}$ in $\{1,\ldots,n\}$ let me write $$ A(P)_{i_1,i_2,\ldots,i_{2q}}=\prod_{\{\alpha,\beta\}\in P} \delta_{i_{\alpha},i_{\beta}}\ . $$ One can view $\wedge^q(\wedge^2(\mathbb{R}^n))$ as the real vector space of all arrays $T=(T_{i_1,\ldots,i_{2q}})_{i_1,\ldots,i_{2q}\in\{1,\ldots,n\}}$ which change sign if one exchanges indices within a block of $P_{\ast}$ or if one rigidly exchanges two blocks. Now define the linear forms $L_P$ indexed by Wick contractions given by $$ L_P(T)=\sum_{i_1,\ldots,i_{2q}\in\{1,\ldots,n\}} A(P)_{i_1,i_2,\ldots,i_{2q}} \ T_{i_1,i_2,\ldots,i_{2q}}\ . $$ By the first fundamental theorem of 19th century invariant theory for $K=O(n)$, these form a linearly generating collection for ${\rm Hom}_K(\wedge^q\mathfrak{p},\mathbb{C})$. Of course one can restrict to $P$'s which are "transverse" to $P_{\ast}$, i.e. , such that $P\cap P_{\ast}=\varnothing$. I don't immediately see how to prune the remaining collection to get a basis, since I expect tons of linear relations. The stable case $n\ge 2q$ might be easier to analyze first.

Note that one can reduce the generating set a bit as follows. Instead of Wick contractions (aka perfect matchings) $P$ one can index invariants by integer partitions $\lambda$ of $q$. It one takes $T=A^{(1)}\wedge\cdots\wedge A^{(q)}$ for a bunch of symmetric matrices $A^{(1)},\ldots,A^{(q)}$ then the invariant encoded by $\lambda$ is the complete antisymmetrization of a "manual polarization" of $$ \prod_{a}\ {\rm tr}(A^{\lambda_a})\ . $$ By "manual polarization" I mean replace each $A$ factor by an $A^{(k)}$ so everyone is used exactly once. If one thinks of $P$'s as fixed point free involutions, take the cycle type of the composition $P\circ P_{\ast}$ and divide all the cycle lengths by two (extra care needed if lack of transversality). That gives you the partition $\lambda$. It is easy to see that the invariant $L_{\lambda}$ vanishes unless all the parts $\lambda_a$ are congruent to 1 mod 4. It also vanishes if there are repeated (odd) parts. Indeed if one does a cyclic permutation along a trace of length $\lambda_a$ on gets the identity $L_{\lambda}=(-1)^{\lambda_a-1}L_{\lambda}$ so an even part kills the invariant. For $\lambda_a$ congruent to 3 mod 4 one can do a "mirror symmetry" of the cycle which fixes a vertex. This involves $(\lambda_a-1)/2$ disjoint transpositions which is an odd number. If there are two cycles of length $\lambda_a$ then exchanging them gives $(-1)^{\lambda_a^2}=-1$ since we reduced the discussion to odd parts.

Let $N(q)$ be the number of integer partitions $\lambda$ of $q$ with distinct parts that are congruent to 1 mod 4. The argument I just gave proves the upper bound $$ {\rm dim}(\wedge^q({\rm Sym}^2(\mathbb{R}^n)))^{O(n)}\le N(q)\ . $$ Interestingly, the computations by Vit also give a computer assisted proof that, in fact, equality holds for $q\le 14$ (and $n$ large enough, i.e., in the stable case).

Update: In the trace of five matrices case one can prove non vanishing as follows. The invariant $L_\lambda$ corresponding to $\lambda=(5)$ is $$ L_{\lambda}(T)=\frac{1}{5!}\sum_{\sigma\in\mathfrak{S}_5} \varepsilon(\sigma) {\rm tr}(A^{(\sigma(1))}\cdots A^{(\sigma(5))}) $$ when $T=A^{(1)}\wedge\cdots\wedge A^{(5)}$. Now take the following $5\times 5$ matrices. Let $A^{(1)}_{12}=A^{(1)}_{21}=1$ and all other entries zero. Let $A^{(2)}_{23}=A^{(2)}_{32}=1$ and all other entries zero. Etc. Let $A^{(5)}_{51}=A^{(5)}_{15}=1$ and all other entries zero. Namely, consider the graph given by the cycle $(12345)$. For each edge in the cycle take the adjacency matrix of corresponding one-edge graph. If I did not mess up my computation, the corresponding specialization of $L_{\lambda}$ is equal to $1/12$.

This can be generalized for a partition $\mu$ of $q$, also with distinct parts congruent to 1 mod 4. One then defines an element $T_{\mu}=A^{(1)}\wedge\cdots\wedge A^{(q)}$ as before. Namely, build a graph $G$ with vertex set $\{1,\ldots,q\}$ made of disjoint cycles with lengths given by the parts of $\mu$. For each edge in $G$ build the adjacency matrix of the corresponding one-edge graph to get the $A$ matrices. I think $L_{\lambda}(T_{\mu})$ is an invertible diagonal matrix (with rows and columns indexed by partitions). This immediately implies that $N(q)$ is the exact answer to the OP's question.

Edit: I misread the question and understood $\mathfrak{p}$ as corresponding to antisymetric instead of symmetric matrices. So what immediately follows answers a different question. See the end of this post for a complete solution of the OP's case.

As for a linearly generating set of invariants, it is easy to construct as follows. Let me call a Wick contraction any set partition $P$ of $\{1,2,\ldots, 2q\}$ with blocks of size two. Let me denote by $P_{\ast}$ the special case $$ \{\{1,2\},\{3,4\},\ldots,\{2q-1,2q\}\}\ . $$ For indices $i,j$ in $\{1,\ldots,n\}$, let me use the Kronecker delta notation $\delta_{i,j}$ for the indicator function of the condition $i=j$. For a Wick contraction $P$ and a collection of $2q$ indices $i_1,\ldots,i_{2q}$ in $\{1,\ldots,n\}$, let me write $$ A(P)_{i_1,i_2,\ldots,i_{2q}}=\prod_{\{\alpha,\beta\}\in P} \delta_{i_{\alpha},i_{\beta}}\ . $$ One can view $\wedge^q(\wedge^2(\mathbb{R}^n))$ as the real vector space of all arrays $T=(T_{i_1,\ldots,i_{2q}})_{i_1,\ldots,i_{2q}\in\{1,\ldots,n\}}$ which change sign if one exchanges indices within a block of $P_{\ast}$ or if one rigidly exchanges two blocks. Now define the linear forms $L_P$ indexed by Wick contractions given by $$ L_P(T)=\sum_{i_1,\ldots,i_{2q}\in\{1,\ldots,n\}} A(P)_{i_1,i_2,\ldots,i_{2q}} \ T_{i_1,i_2,\ldots,i_{2q}}\ . $$ By the first fundamental theorem of 19th century invariant theory for $K=O(n)$, these form a linearly generating collection for ${\rm Hom}_K(\wedge^q\mathfrak{p},\mathbb{C})$. Of course one can restrict to $P$'s which are "transverse" to $P_{\ast}$, i.e. , such that $P\cap P_{\ast}=\varnothing$. I don't immediately see how to prune the remaining collection to get a basis, since I expect tons of linear relations. The stable case $n\ge 2q$ might be easier to analyze first.

Note that one can reduce the generating set a bit as follows. Instead of Wick contractions (aka perfect matchings) $P$ one can index invariants by integer partitions $\lambda$ of $q$. If one takes $T=A^{(1)}\wedge\cdots\wedge A^{(q)}$ for a bunch of symmetric matrices $A^{(1)},\ldots,A^{(q)}$ then the invariant encoded by $\lambda$ is the complete antisymmetrization of a "manual polarization" of $$ \prod_{a}\ {\rm tr}(A^{\lambda_a})\ . $$ By "manual polarization" I mean replace each $A$ factor by an $A^{(k)}$ so everyone is used exactly once. If one thinks of $P$'s as fixed-point-free involutions, take the cycle type of the composition $P\circ P_{\ast}$ and divide all the cycle lengths by two (extra care needed if lack of transversality). That gives you the partition $\lambda$. It is easy to see that the invariant $L_{\lambda}$ vanishes unless all the parts $\lambda_a$ are congruent to 1 mod 4. It also vanishes if there are repeated (odd) parts. Indeed if one does a cyclic permutation along a trace of length $\lambda_a$ one gets the identity $L_{\lambda}=(-1)^{\lambda_a-1}L_{\lambda}$. So an even part kills the invariant. For $\lambda_a$ congruent to 3 mod 4 one can do a "mirror symmetry" of the cycle which fixes a vertex. This involves $(\lambda_a-1)/2$ disjoint transpositions which is an odd number. If there are two cycles of length $\lambda_a$ then exchanging them gives $(-1)^{\lambda_a^2}=-1$ since we reduced the discussion to odd parts.

Let $N(q)$ be the number of integer partitions $\lambda$ of $q$ with distinct parts that are congruent to 1 mod 4. The argument I just gave proves the upper bound $$ {\rm dim}(\wedge^q({\rm Sym}^2(\mathbb{R}^n))^{O(n)})\le N(q)\ . $$ Interestingly, the computations by Vit also give a computer assisted proof that, in fact, equality holds for $q\le 14$ (and $n$ large enough, i.e., in the stable case).

Update: In the trace of five matrices case one can prove nonvanishing as follows. The invariant $L_\lambda$ corresponding to $\lambda=(5)$ is $$ L_{\lambda}(T)=\frac{1}{5!}\sum_{\sigma\in\mathfrak{S}_5} \varepsilon(\sigma)\ {\rm tr}(A^{(\sigma(1))}\cdots A^{(\sigma(5))}) $$ when $T=A^{(1)}\wedge\cdots\wedge A^{(5)}$. Now take the following $5\times 5$ matrices. Let $A^{(1)}_{12}=A^{(1)}_{21}=1$ and all other entries zero. Let $A^{(2)}_{23}=A^{(2)}_{32}=1$ and all other entries zero. Etc. Let $A^{(5)}_{51}=A^{(5)}_{15}=1$ and all other entries zero. Namely, consider the graph given by the cycle $(12345)$. For each edge in the cycle take the adjacency matrix of the corresponding one-edge graph. If I did not mess up my computation, the corresponding specialization of $L_{\lambda}$ is equal to $1/12$.

This can be generalized for a partition $\mu$ of $q$, also with distinct parts congruent to 1 mod 4. One then defines an element $T_{\mu}=A^{(1)}\wedge\cdots\wedge A^{(q)}$ as before. Namely, build a graph $G$ with vertex set $\{1,\ldots,q\}$ made of disjoint cycles with lengths given by the parts of $\mu$. For each edge in $G$ build the adjacency matrix of the corresponding one-edge graph to get the $A$ matrices. If one has a part $\mu_a=1$ corresponding to a cycle $(j)$ then take the matrix with $A_{jj}=1$ and all other entries zero. A quick combinatorial argument shows that $L_{\lambda}(T_{\mu})$ is an invertible diagonal matrix (with rows and columns indexed by partitions). By construction products of $A$'s give zero most of the time unless $\lambda$ replicates $\mu$. This immediately implies that $N(q)$ is the exact answer to the OP's question.

Edit: I misread the question and understood $\mathfrak{p}$ as corresponding to antisymetric instead of symmetric matrices. So what immediately follows answers a different question. See the end of this answer for OP's case.

I don't know what you mean exactly by "compute the invariants": perhaps finding an explicit linear basis? Essentially $\wedge^q\mathfrak{p}$ is the same as the antisymmetric plethysm $\wedge^q(\wedge^2(\mathbb{R}^n))$. If I remember correctly this kind of objects appears in relation to the Hodge conjecture for certain Abelian varieties as in work by Ken Ribet and Fumio Hazama. Namely, in some particular situations, the invariant elements in such a plethysm are the Hodge classes in the cohomology of the variety. See in particular the article "Branching rules and Hodge cycles on certain Abelian varieties" in AJM 1988. There could be some useful tools for your question in that literature.

Update: In the trace of five matrices case one can prove non vanishing as follows. The invariant $L_\lambda$ corresponding to $\lambda=(5)$ is $$ L_{\lambda}(T)=\frac{1}{5!}\sum_{\sigma\in\mathfrak{S}_5} \varepsilon(\sigma) {\rm tr}(A^{(\sigma(1))}\cdots A^{(\sigma(5))}) $$ when $T=A^{(1)}\wedge\cdots\wedge A^{5}$. Now take the following $5\times 5$ matrices. Let $A^{(1)}_{12}=A^{(1)}_{21}=1$ and all other entries zero. Let $A^{(2)}_{23}=A^{(2)}_{32}=1$ and all other entries zero. Etc. Let $A^{(5)}_{51}=A^{(5)}_{15}=1$ and all other entries zero. Namely, consider the graph given by the cycle $(12345)$. For each edge in the cycle take the adjacency matrix of corresponding one-edge graph. If I did not mess up my computation, the corresponding specialization of $L_{\lambda}$ is equal to $1/12$.

This can be generalized for a partition $\mu$ of $q$, also with distinct parts congruent to 1 mod 4. One then defines an element $T_{\mu}=A^{(1)}\wedge\cdots\wedge A^{(q)}$ as before. Namely, build a graph $G$ with vertex set $\{1,\ldots,q\}$ made of disjoint cycles with lengths given by the parts of $\mu$. For each edge in $G$ build the adjacency matrix of the corresponding one-edge graph to get the $A$ matrices. I think $L_{\lambda}(T_{\mu})$ is an invertible diagonal matrix. This would immediately imply that $N(q)$ is the exact answer to the OP's question.

Edit: I misread the question and understood $\mathfrak{p}$ as corresponding to antisymetric instead of symmetric matrices. So what immediately follows answers a different question. See the end of this answer for a complete solution of the OP's case.

I don't know what you mean exactly by "compute the invariants": perhaps finding an explicit linear basis? Essentially $\wedge^q\mathfrak{p}$ is the same as the antisymmetric plethysm $\wedge^q(\wedge^2(\mathbb{R}^n))$. If I remember correctly, this kind of objects appears in relation to the Hodge conjecture for certain Abelian varieties as in work by Ken Ribet and Fumio Hazama. Namely, in some particular situations, the invariant elements in such a plethysm are the Hodge classes in the cohomology of the variety. See in particular the article "Branching rules and Hodge cycles on certain Abelian varieties" in AJM 1988. There could be some useful tools for your question in that literature.

Update: In the trace of five matrices case one can prove non vanishing as follows. The invariant $L_\lambda$ corresponding to $\lambda=(5)$ is $$ L_{\lambda}(T)=\frac{1}{5!}\sum_{\sigma\in\mathfrak{S}_5} \varepsilon(\sigma) {\rm tr}(A^{(\sigma(1))}\cdots A^{(\sigma(5))}) $$ when $T=A^{(1)}\wedge\cdots\wedge A^{(5)}$. Now take the following $5\times 5$ matrices. Let $A^{(1)}_{12}=A^{(1)}_{21}=1$ and all other entries zero. Let $A^{(2)}_{23}=A^{(2)}_{32}=1$ and all other entries zero. Etc. Let $A^{(5)}_{51}=A^{(5)}_{15}=1$ and all other entries zero. Namely, consider the graph given by the cycle $(12345)$. For each edge in the cycle take the adjacency matrix of corresponding one-edge graph. If I did not mess up my computation, the corresponding specialization of $L_{\lambda}$ is equal to $1/12$.

This can be generalized for a partition $\mu$ of $q$, also with distinct parts congruent to 1 mod 4. One then defines an element $T_{\mu}=A^{(1)}\wedge\cdots\wedge A^{(q)}$ as before. Namely, build a graph $G$ with vertex set $\{1,\ldots,q\}$ made of disjoint cycles with lengths given by the parts of $\mu$. For each edge in $G$ build the adjacency matrix of the corresponding one-edge graph to get the $A$ matrices. I think $L_{\lambda}(T_{\mu})$ is an invertible diagonal matrix (with rows and columns indexed by partitions). This immediately implies that $N(q)$ is the exact answer to the OP's question.

Update: In the trace of five matrices case one can prove non vanishing as follows. The invariant $L_\lambda$ corresponding to $\lambda=(5)$ is $$ L_{\lambda}(T)=\frac{1}{5!}\sum_{\sigma\in\mathfrak{S}_5} \varepsilon(\sigma) {\rm tr}(A^{(\sigma(1))}\cdots A^{(\sigma(5))}) $$ when $T=A^{(1)}\wedge\cdots\wedge A^{5}$. Now take the following $5\times 5$ matrices. Let $A^{(1)}_{12}=A^{(1)}_{21}=1$ and all other entries zero. Let $A^{(2)}_{23}=A^{(2)}_{32}=1$ and all other entries zero. Etc. Let $A^{(5)}_{51}=A^{(5)}_{15}=1$ and all other entries zero. Namely, consider the graph given by the cycle $(12345)$. For each edge in the cycle take the adjacency matrix of corresponding one-edge graph. If I did not mess up my computation, the corresponding specialization of $L_{\lambda}$ is equal to $1/12$.

This can be generalized for a partition $\mu$ of $q$, also with distinct parts congruent to 1 mod 4. One then defines an element $T_{\mu}=A^{(1)}\wedge\cdots\wedge A^{(q)}$ as before. Namely, build a graph $G$ with vertex set $\{1,\ldots,q\}$ made of disjoint cycles with lengths given by the parts of $\mu$. For each edge in $G$ build the adjacency matrix of the corresponding one-edge graph to get the $A$ matrices. I think $L_{\lambda}(T_{\mu})$ is an invertible diagonal matrix. This would immediately imply that $N(q)$ is the exact answer to the OP's question.