# Self-normalizing implies maximal for subgroup of compact Lie group

Consider the compact group $$G=\operatorname{SO}_3(\mathbb{R})$$. The closed subgroups of $$G$$ (other than the trivial group 1 and the whole group $$G$$) are exactly $$O_2$$, $$\operatorname{SO}_2$$ and the finite groups $$C_n$$, $$D_{2n}$$, $$T \cong A_4$$, $$O \cong S_4$$, $$I \cong A_5$$ (cyclic groups with $$n$$ elements, dihedral groups with $$2n$$ elements and the three symmetry groups of the platonic solids). The normalizers of these groups are as follows:

\begin{align*} G&=N_G(G)=N_G(1) \\ O_2&=N_G(O_2)=N_G(\operatorname{SO}_2)=N_G(C_n) \\ I&= N_G(I) \\ O&=N_G(O)=N_G(T)=N_G(D_4) \\ D_{4n} &= N_G(D_{2n}) \end{align*} where in the last equation $$n \geq 3$$. We say a (closed) subgroup is maximal if it is maximal among all proper closed subgroups of $$G$$.

Observe that in the example above the maximal subgroups exactly coincide with the self-normalizing subgroups. Namely, $$O_2, I,O.$$ That the maximal subgroups are all self-normalizing is not too surprising. The normalizer of a closed subgroup is always closed. Thus a maximal subgroup is always either normal or self-normalizing. Since $$G$$ is simple, adjoint (i.e. center-free), and connected that means the maximal subgroups must be self-normalizing. However I am a bit surprised that the reverse holds. That is, that every self-normalizing subgroup of $$G$$ is maximal. That inspires my question:

For a closed subgroup of a compact Lie group does self-normalizing imply maximal?

This is true for the compact Lie group $$\operatorname{SO}_3(\mathbb{R})$$ and thus also true for $$\operatorname{SU}_2$$. What about the generic case? I am especially interested in $$\operatorname{SU}_3$$.

• Do you mean to restrict to compact groups? Any parabolic subgroup is self-normalising, and there are plenty of non-maximal parabolics in large-rank non-compact groups. (By the way, normalisers are always closed, whether of closed or non-closed subgroups, or even of arbitrary subsets.) Jan 15, 2022 at 19:46
• Yes indeed I want to restrict to compact groups! I'll edit accordingly. As you say non compact groups have lots of self normalizing subgroups. Including all the parabolic subgroups. For example the subgroup of upper triangular matrices of $SL_3(\mathbb{C})$ (the Borel subgroup) is self normalizing but not maximal. Jan 15, 2022 at 22:35
• @LSpice about the normalisers I'm not so sure. Don't you need the subset/subgroup to be closed? For example consider the non closed subgroup $H=O_2(\mathbb{Q})$ of $G=O_2(\mathbb{R})$. I think $H$ is self normalizing and thus not closed normalizer Jan 16, 2022 at 5:09
• I don't know whether your $H$ is self normalising, but, you're right, my statement that normalisers are always closed was too hasty. Jan 16, 2022 at 15:50
• Ok you're right it's not self normalizing. The normalizer is in fact the group generated by (a rational reflection and) all rotations that square to a rational rotation. For example the normalizer contains a rotation by $\pi/4$. However the normalizer is still not closed and fails to contain things like rotation by $\pi/3$ Jan 16, 2022 at 18:09

$$\DeclareMathOperator\SO{SO}\DeclareMathOperator\U{U}\DeclareMathOperator\O{O}$$Denote by $$\U'(n)$$ the normalizer of $$\U(n)$$ in $$\mathrm{GL}_{2n}(\mathbf{R})$$. It is not hard to see that $$\U(n)$$ has index 2 in $$\U'(n)$$, which is generated by $$\U(n)$$ and by the coordinate-wise complex conjugation. Moreover, $$\U'(n)$$ is maximal in $$\O(2n)$$ (if $$n\ge 2$$).

In $$G=\SO(5)$$, consider the subgroup $$H=\SO(5)\cap (\U'(2)\times \O(1))$$ (which contains $$\U(2)$$ with index 2). I claim that $$H$$ is self-normalized, but not maximal, in $$G=\SO(5)$$.

The subgroup $$H$$ is not maximal in $$G$$ because it is properly contained in $$L=SO(5)\cap (\O(4)\times \O(1))$$.

It is self-normalized. Indeed, its action on $$\mathbf{R}^5$$ has the irreducible decomposition $$4\oplus 1$$, which is preserved by the normalizer. Hence, the normalizer of $$H$$ in $$G$$ equals the normalizer of $$H$$ inside $$\SO(5)\cap (\O(4)\times \O(1))=L$$. Using that $$\U'(2)$$ is maximal in $$\O(4)$$ one can deduce easily that $$H$$ is maximal in $$L$$. Since $$H$$ is not normal in $$L$$, it is self-normalized in $$L$$. Hence $$H$$ is self-normalized in $$G$$.

• Should the action of $L$ be on $\mathbb R^5$, not on $\mathbb R^4$? Otherwise I do not see how the action works, or understand how it can decompose as $4 + 1$. (Actually I don't know what $4$ and $1$ mean, since I do not know how to label representations of non-rank-$1$ groups by integers, but at least it seems like $4 + 1$ should be $5$-dimensional.) Jan 16, 2022 at 15:51
• @LSpice yes it was a typo. Yes I ignored the proper labeling, by labeling by dimensions, which is not enough in general but enough here.
– YCor
Jan 16, 2022 at 17:19
• Awesome example! Would you happen to know an example with a similar setup (a simple compact group with a self-normalizing but not maximal subgroup) where the subgroup in question is also finite? Because I'm still curious if self-normalizing might be equivalent to maximal for finite subgroups of compact simple groups. Jan 17, 2022 at 0:16
• I'm pretty sure there are finite examples as well. I just don't have in mind what kind of finite subgroups occur as maximal (closed proper) subgroups in $\mathrm{SO}(n)$ for, say $n=5$, or in $\mathrm{SU}(3)$.
– YCor
Jan 17, 2022 at 8:38
• Yes I am very interested in maximal closed proper subgroups of $SU_3$. A central extension of $A_6$ by cyclic 3 is one such example. Also I think central extension of $PSL(2,7)$ by cyclic 3. There are more details and references in this fabulous MO post mathoverflow.net/questions/17072/the-finite-subgroups-of-sun Jan 20, 2022 at 4:35