Orthogonal transformations fixing a subspace (setwise)

Question

Let $(V,Q)$ be a non-degenerate quadratic space of dimension $n$ over an algebraically closed field of characteristic zero. Let $W$ be a subspace of $V$ of dimension $m < \frac12 n$ which is totally isotropic for $Q$. What is the structure of the subgroup of $O(Q)$ consisting of orthogonal transformations of $(V,Q)$ which send $W$ to itself? I'm looking for some (sort of) concrete description...

Answer 1

The full orthogonal group $O(Q)$ is generated by reflections, i.e. involutory isometries fixing a hyperplane pointwise: $$\pi_v : V \to V : x \mapsto x - \frac{Q(v,x)}{Q(v)} v,$$ where $v$ is an anisotropic vector orthogonal to the hyperplane. (So we only consider those hyperplanes for which the normal vector is indeed anisotropic.)

The pointwise stabilizer of a given subspace $W$ is then the subgroup generated by those reflections corresponding to the hyperplanes containing $W$. If you want to obtain the elements of $O(Q)$ only stabilizing $W$ setwise (which is what you are asking), then in addition you have to take the elements of the subgroup $O(Q_{|W})$ into account.

I'm not sure whether this description is "concrete" in the sense that you are looking for. Anyhow, it does not use any of your assumptions "algebraically closed", "characteristic zero" or "$W$ totally isotropic", but let's assume that the base field is not $\mathbb{F}_2$ to be on the safe side.

Answer 2

I posted this answer over at Math Stack Exchange, but since I'm not too sure about it I thought I should post it here as well. Hopefully someone who knows more about this than I do can look it over and check whether it's right.

I claim that the following statements hold for any subspace $W$ of $V$ (not just totally isotropic spaces). In the totally isotropic case, $W = R$ and $W^\perp = S$, which makes the solution slightly simpler.

Let $W$ be an arbitrary subspace of $V$, and let $G \leq O(V)$ be the group of orthogonal transformations that leave $W$ invariant. Define $$ R \;=\; W \cap W^\perp \qquad\text{and}\qquad S \;=\; W + W^\perp. $$ Note that $W^\perp$ is invariant under the action of $G$, and therefore $R$ and $S$ are also invariant.

The structure of $G$ is as follows. First, there is a short exact sequence $$ 1 \;\;\to\;\; A \;\;\to\;\; G \;\;\to\;\; O(S) \;\;\to\;\; 1 $$ where $A$ is an abelian group isomorphic to $K^d$ for some value of $d$. The homomorphism $G \to O(S)$ is surjective because of Witt's theorem.

The group $O(S)$ is a semidirect product. Specifically, $$ O(S) \;\cong\; \bigl( \text{Lin}(R,W/R) \times \text{Lin}(R,W^\perp/R) \bigr) \;\rtimes\; \bigl(O(R) \times O(W/R) \times O(W^\perp/R) \bigr). $$ Here $\text{Lin}(R,W/R)$ is the additive abelian group of all linear functions $R\to W/R$, and $\text{Lin}(R,W^\perp/R)$ is similarly an additive abelian group. Since $Q$ restricts to the null quadratic form on $R$, the orthogonal group $O(R)$ is the same as $GL(R)$. Moreover, since $Q$ is null on $R$, the quotients $W/R$ and $W^\perp/R$ are quadratic spaces, and $O(W/R)$ and $O(W^\perp/R)$ are the corresponding orthogonal groups. Note also that the quadratic forms on $W/R$ and $W^\perp/R$ are nondegenerate.

Edit: Here is a bit more information on the kernel $A$ of the epimorphism $G \to O(S)$. Since $Q|_{R\times S} = 0$, the quadratic form $Q$ defines a bilinear map $B \colon R \times (V/S) \to K$, and it is not hard to show that $B$ is a perfect pairing. It follows that the action of an element $g\in G$ on $V/S$ is entirely determined by the action of $g$ on $R$. In particular, every element of $A$ acts trivially on $V/S$. Therefore, every element $g\in A$ has the form $$ g(v) \;=\; v + \varphi(\pi(v)) $$ where $\pi\colon V \to V/S$ is the quotient map, and $\varphi\colon V/S \to S$ is a linear map. Thus $A$ is isomorphic to some subgroup of the abelian group $\text{Lin}(V/S,S)$.

To be specific, $A$ is isomorphic to the group of all linear maps $\varphi\colon V/S \to S$ that satisfy the following conditions:

1. The range of $\varphi$ lies in $R$.

2. The map $\varphi$ is "antisymmetric" with respect to $B$ in the sense that $$ B(\varphi(u),v) + B(\varphi(v),u) = 0 $$ for all $u,v \in V/S$.

In particular, $A$ is isomorphic to the additive group of all $m\times m$ antisymmetric matrices over $K$, where $m = \dim(R)$.