Maximal compact subgroup of $\mathrm{SL}(2,\mathbb{H})$

Question

$\DeclareMathOperator\Spin{Spin}\DeclareMathOperator\Sp{Sp}\DeclareMathOperator\SL{SL}$The exceptional isomorphism $\Spin(5,1)\simeq \SL(2,\mathbb{H})$ is well-known, and I can find references that say the maximal compact of $\Spin(5,1)$ is $\Spin(5) \simeq \Sp(2)$. So I know the answer to the question, but not the how or why. In particular, is there a proof that $\Sp(2)$ is maximal compact in $\SL(2,\mathbb{H})$ not via the exceptional isomorphisms? Perhaps some sort of analogue of Gram-Schmidt or other explicit factorisation?

Answer 1

$\DeclareMathOperator\GL{GL}\DeclareMathOperator\SL{SL}\DeclareMathOperator\SO{SO}\DeclareMathOperator\Sp{Sp}$YCor's comment contains the essential idea needed for the proof, but maybe a few more details would be helpful. (If YCor does provide something similar later, feel free to award YCor's answer the bounty.)

Consider the mapping $\sigma:\GL(2,\mathbb{H})\to M_2(\mathbb{H})$ given by $$ \sigma(A) = A^* A $$ where $A^*$ is the conjugate transpose of $A$ in $M_2(\mathbb{H})$, the $2$-by-$2$ matrices with entries in $\mathbb{H}$. Then $\sigma(A)=\sigma(A)^*$ (using the fact that $\overline{pq}= \overline{q}\,\overline{p}$ for $p,q\in\mathbb{H}$). Consequently, the image of $\sigma$ lies in the $6$-dimensional real subspace $S_2(\mathbb{H})$, consisting of the matrices $s\in M_2(\mathbb{H})$ that satisfy $s = s^*$. Due to the associativity of multiplication in $\mathbb{H}$ and the above-mentioned conjugation identity, we have $$ (AB)^*sAB = B^*(A^*sA)B, $$ so it follows that the mapping $\rho(s,A)=A^*sA$ defines a (right) representation of $\GL_2(\mathbb{H})$ on $S_2(\mathbb{H})$.

Now, define the quadratic form $Q:S_2(\mathbb{H})\to\mathbb{R}$ by $$ Q\left(\begin{pmatrix}a&x\\\overline{x}&b\end{pmatrix}\right) = ab-x\overline{x}. $$ when $a,b\in\mathbb{R}$ and $x\in\mathbb{H}$. Note that $Q$ has signature type $(1,5)$ as a real quadratic form.

The crucial identity (which can be proved by hand just by writing it out) is that $$ Q(A^*sA) = Q(s)\,Q(A^*A). $$ It follows that $Q(A^*A) = 1$ defines $\SL(2,\mathbb{H})$ as a codimension $1$ closed subgroup of $\GL(2,\mathbb{H})$. In particular, the representation $\rho$ sends $\SL(2,\mathbb{H})$ into $\SO(Q)\simeq\SO(1,5)$, and it is easy to show that the kernel of this homomorphism is $\{\pm I_2\}\subset\SL(2,\mathbb{H})$.

By definition, the stabilizer of $I_2$ under the right representation $\rho$ is the 10-dimensional Lie group usually denoted in differential geometry by $\Sp(2)$, and its orbit under this right action by $\SL(2,\mathbb{H})$ must thus have dimension $5$ and hence be the nappe of the hyperboloid $Q(s)=1$ consisting of those $s$ with positive trace, i.e., hyperbolic $5$-space.

The connectedness of $\Sp(2)$ follows from the well-known fibration $\Sp(1)\to \Sp(2)\to S^7$, so it follows that $\Sp(2)$ is a nontrivial double cover of the identity component of $\SO(Q)$. Since $\SL(2,\mathbb{H})$ is simple and not compact, the signature type of its Killing form cannot be $(0,15)$, and since its Lie algebra splits as a module over ${\mathfrak{sp}}(2)$ into two irreducible pieces of dimension $10$ and $5$ corresponding to ${\mathfrak{sp}}(2)$ (on which it is negative definite) and its orthogonal complement, the type must be $(5,10)$. Thus, a maximal compact in $\SL(2,\mathbb{H})$ must have dimension $10$, so $\Sp(2)$ must be a maximal compact.

Finally, if you want a ‘factorization’ (analogous to the QR decomposition in the real case), you can show that every $A\in\SL(2,\mathbb{H})$ can be factored uniquely in the form $$ A = Q\begin{pmatrix} a & 0\\ 0& a^{-1}\end{pmatrix}\begin{pmatrix} 1 & x\\ 0& 1\end{pmatrix}, $$ with $Q\in\Sp(2)$, $a\in\mathbb{R}^+$, and $x\in\mathbb{H}$, which also shows that $\SL(2,\mathbb{H})$ is diffeomorphic to $\Sp(2)\times\mathbb{R}^5$. (This is just the KAN decomposition for $\SL(2,\mathbb{H})$.)

Answer 2

Here is a different argument from Robert's.

Observation: given a compact Lie group $G$ and a proper closed subgroup $H$ the inclusion $H\to G$ is not a homotopy equivalence, else the homogeneous space $G/H$ would be a closed contractible manifold.

Next, it's a general fact that the inclusion of a maximal compact subgroup $K$ into a semisimple Lie group $G$ is a homotopy equivalence and the quotient space is contractible. Putting these two things together immediately implies that if $G$ is a semisimple Lie group and $H\subset G$ is a compact subgroup such that $G/H$ is contractible then $H$ is maximal compact in $G$.

It remains to observe that $SL(2,\mathbb H)/Sp(2)$ is contractible.

From Robert's answer the homogeneous (in fact, symmetric) space $SL(2,\mathbb H)/Sp(2)$ can be identified with the 5-dimensional hyperbolic space which is of course contractible.

Alternatively this can also be seen as follows.

Look at the standard transitive action of $Sp(2)$ on the unit sphere $S^7$ in $\mathbb H^2$. The stabilizer of $(1,0)$ is easily seen to be equal to $Sp(1)=S^3$.

Similarly look at the action of $SL(2,\mathbb H)$ on $\mathbb H^2\setminus (0,0)$ ( which deformation retracts onto $S^7$). the stabilizer $K$ of $(1,0)$ consists of matrices of the form $\begin{pmatrix} 1&a\\0&b\end{pmatrix}$ where $|b|=1$ and $a\in \mathbb H$ is arbitrary. So $K=S^3\times \mathbb R^4$ topologically.

Comparing long exact homotopy sequences of the homogeneous space bundles $Sp(1)\to Sp(2)\to Sp(2)/Sp(1)=S^7$ and $K\to SL(2,\mathbb H)\to SL(2,\mathbb H)/K=\mathbb H^2\setminus (0,0)$ and using the natural map between them it now immediately follows by the 5-lemma that the inclusion $Sp(2)\subset SL(2,\mathbb H)$ is a homotopy equivalence. Hence by above $Sp(2)$ is maximal compact in $SL(2,\mathbb H)$.

By induction this generalizes to show that $Sp(n)$ is maximal compact in $SL(n,\mathbb H)$.