# How to detect if a subgroup lands inside an orthogonal group?

Equivalently, my question may be phrased as, "Are there defining characteristics of representations of orthogonal (symmetric form-preserving) groups?"

Here I am working with a unitary representation of subgroups of the reductive group $GL(n, \mathbb{C})$, and I am seeking a criterion which I may apply to representations of arbitrary $H \leq GL(n, \mathbb{C})$.

For $(V, \rho)$ a representation of $H$ such that $H \leq O(n, \Bbb C)$, one such criterion that I had thought may be true is that this representation must necessarily be self-dual i.e., isomorphic to the dual representation. Is this in fact the case?

Any references would be appreciated. I am currently reading Dynkin's "Maximal Subgroups of the Classical Groups" in search of a lead. Many thanks for taking the time to read this.

-
Yes, of course that is the case. The natural representation of the orthogonal group is self dual,hence so is its restriction to $H$. But not conversely: the symplectic group also has self dual natural representation. –  Venkataramana May 18 '13 at 15:03
There are criteria to tell if the form that the "self-dual" group $H$preserves is orthogonal or symplectic, in terms of the action by a particular element of the centre of $H$. All this is classical, and I am sure is in standard textbooks. –  Venkataramana May 18 '13 at 15:05
@Aakumadula thank you. I will consult Fulton and Harris, I will surely find it in there –  KingOliver May 18 '13 at 15:13
Also note that when $n$ is even, $GL(n/2,\mathbb{C})$ can be embedded in $O(n,\mathbb{C})$. –  Name Jan 9 '14 at 10:18

This answer may be more general than you need. Let $F$ be a field, and let $G=\langle g_1,\dots,g_m\rangle$ be a subgroup of ${\rm GL}(n,F)$ which preserves a form, with matrix $J$. Then $g_i J g_i^{\rm T}=J$ holds for $i=1,\dots,m$. Assume the form is nondegenerate, and hence $J$ is invertible. Finding whether or not there exists an invertible $J$ reduces to solving the equations $J^{-1}g_iJ=g_i^{\rm -T}$ for $i=1,\dots,m$. This relates the natural embedding of $G$ to its contragredient. The form could be symplectic, unitary, or orthogonal (in characteristic $\ne 2$).
Finding $J$ is easiest when $G$ acts absolutely irreducibly on the vector space $F^n$. A tedious method for large $n$ involves linear algebra: view the $n^2$ entries of $J$ as unknowns and solve the $mn^2$ linear equations $g_i J = Jg_i^{\rm -T}$ where $i=1,\dots,m$. A solution, when $G$ is absolutely irreducible, is unique up to a scalar multiple. By Schur's lemma $J$ is either $0$, or invtertible. A much faster probabilistic algorithm, called the Meat-Axe, finds an invertible $J$, or proves that no such $J$ exists. (More can be said if $G$ preserves a form up to scalars, or $G$ preserves a quadratic form and ${\rm char}(F)=2$, but here is not the place for that. The Meat-Axe relies on finding "good" matrices whose densities are not well understood when $F$ is infinite. The algorithm always gives the correct answer, but it may not terminate if it fails to find a "good" matrix.)