Counting adjoints in the symmetric or antisymmetric square of a Lie group representation EDIT (November 1, 2022): Over the weekend I think I found a technique to determine the exact multiplicities, according to how conjugation acts on the fundamental weights.  While I haven't done the precise counting yet, the inequalities I was suggesting here are indeed correct.  I plan to write it up with a colleague in the next couple of weeks and post an answer to this question here as soon as I can manage.
Consider an irreducible representation $R$ of a simple compact Lie group $G$, such that $R$ is isomorphic to its dual.  This isomorphism (unique up to scaling by Schur's lemma) defines a trivial one-dimensional subrepresentation of $R\otimes R=S^2R\oplus\Lambda^2R$ and the representation is called real/quaternionic if $S^2R$ (resp $\Lambda^2R$) contains this one-dimensional representation of $G$.
I am interested in the multiplicities $p_S$ and $p_\Lambda$ with which the adjoint representation of $G$ appears in the decompositions of $S^2R$ and $\Lambda^2R$ into irreducible representations.  Experimentally, I observed that

*

*$p_\Lambda \geq p_S$ for a real representation,


*$p_S > p_\Lambda$ for a quaternionic representation.
Is it true?  Here are some useless facts I have derived so far.

*

*For a real representation, $p_\Lambda\geq 1$.  For a quaternionic representation, $p_S\geq 1$.  (This comes from looking at the action of $\text{Lie}(G)$ on $R$ as an element of $\text{Hom}(\text{adj}_G\otimes R,R) \simeq \text{Hom}(\text{adj}_G, R\otimes R)$ and tracking symmetries.)


*$p=p_\Lambda+p_S$ is equal to the number of simple roots $\alpha$ of $G$ such that $\langle\alpha,\mu\rangle>0$, where $\mu$ is the highest weight of $R$.  In other words it is equal to the rank of $G$ minus the number of boundaries of the Weyl chamber that $\mu$ sits on.  Any counterexample must have $p\geq 2$, which makes it a bit hard to scan for counterexamples.


*The question can be reformulated in terms of the Frobenius-Schur indicator and a variant involving the character $\chi_{\text{adj}}(g)$, namely I claim that $\int_{g\in G}\chi(g^2)d\mu$ and $\int_{g\in G}\chi_{\text{adj}}(g) \chi(g^2)d\mu$ have opposite sign, where $\mu$ is the Haar measure.
This came up in the course of writing an article on supersymmetric gauge theories.  I can weaken our conclusions slightly to avoid needing this result, but I would prefer to find a proof or counterexample.
 A: I don't have a proof but I noticed some things while having a quick look for some counterexamples that I thought were worth sharing. First things first what you are looking for is not special to the compact Lie group but can be seen in its complexification. Indeed you are just looking at the self-dual representations of $G^{\mathbb{C}}$ (or even more easily $\mathfrak{g}^\mathbb{C}$) which must have either a invariant symmetric bilinear form or an invariant symplectic form. It so happens that such a symmetric form tells you the corresponding representation of compact $G$ is real and a symplectic form tells you it is quaternionic.
While searching through a bunch of examples using LiE, I couldn't find any counterexamples to your claim. I did however notice some stronger patterns. For groups of type $B,C,D,F,G$ it seems that $p_S =0$ for a real representation and  $p_\Lambda = 0$ for a quaternionic representation. I don't know if this holds for every possible representation but if true it leaves only the $A$ and $E$ types to prove your statements for.
In case they are useful these are the functions I used to hunt for examples in LiE:
v_coef(pol p; vec v) = loc i = 1; loc m = 0; while i<= length(p) do if expon(p, i) == v then m = coef(p,i); break else i = i+1; fi od; m

alt_sym_compare(vec v) = s = sym_tensor(2,v); a =alt_tensor(2,v); t = null(Lie_rank); if v_coef(s,t) > 0 then print(“Real”) fi; if v_coef(a,t) > 0 then print(“Quaternionic”) fi; print(“adjoint in sym square = “); print(v_coef(s,expon(adjoint,1))); print(“adjoint in alt square = “); print(v_coef(a,expon(adjoint,1)))

v_coef finds the multiplicity of a given irrep (given as a vector w.r.t. the fundamental weights) in a representation (given by its characteristic polynomial).
alt_sym_compare uses this to find the multiplicities of the trivial and adjoint representations in $S^2 R$ and $\Lambda^2 R$ and prints out the pertinent information.
These make scanning for counterexamples relatively painless but will struggle with really big representations (I couldn't think of a way to do this without actually computing $S^2 R$ and $\Lambda^2 R$ which can get very complicated)
