How to detect if a subgroup lands inside an orthogonal group?

Question

Equivalently, my question may be phrased as, "Are there defining characteristics of representations of orthogonal (symmetric form-preserving) groups?"

Here I am working with a unitary representation of subgroups of the reductive group $GL(n, \mathbb{C})$, and I am seeking a criterion which I may apply to representations of arbitrary $H \leq GL(n, \mathbb{C})$.

For $(V, \rho)$ a representation of $H$ such that $H \leq O(n, \Bbb C)$, one such criterion that I had thought may be true is that this representation must necessarily be self-dual i.e., isomorphic to the dual representation. Is this in fact the case?

Any references would be appreciated. I am currently reading Dynkin's "Maximal Subgroups of the Classical Groups" in search of a lead. Many thanks for taking the time to read this.

Answer 1

This answer may be more general than you need. Let $F$ be a field, and let $G=\langle g_1,\dots,g_m\rangle$ be a subgroup of ${\rm GL}(n,F)$ which preserves a form, with matrix $J$. Then $g_i J g_i^{\rm T}=J$ holds for $i=1,\dots,m$. Assume the form is nondegenerate, and hence $J$ is invertible. Finding whether or not there exists an invertible $J$ reduces to solving the equations $J^{-1}g_iJ=g_i^{\rm -T}$ for $i=1,\dots,m$. This relates the natural embedding of $G$ to its contragredient. The form could be symplectic, unitary, or orthogonal (in characteristic $\ne 2$).

Finding $J$ is easiest when $G$ acts absolutely irreducibly on the vector space $F^n$. A tedious method for large $n$ involves linear algebra: view the $n^2$ entries of $J$ as unknowns and solve the $mn^2$ linear equations $g_i J = Jg_i^{\rm -T}$ where $i=1,\dots,m$. A solution, when $G$ is absolutely irreducible, is unique up to a scalar multiple. By Schur's lemma $J$ is either $0$, or invtertible. A much faster probabilistic algorithm, called the Meat-Axe, finds an invertible $J$, or proves that no such $J$ exists. (More can be said if $G$ preserves a form up to scalars, or $G$ preserves a quadratic form and ${\rm char}(F)=2$, but here is not the place for that. The Meat-Axe relies on finding "good" matrices whose densities are not well understood when $F$ is infinite. The algorithm always gives the correct answer, but it may not terminate if it fails to find a "good" matrix.)