Self-normalizing implies maximal for subgroup of compact Lie group

Question

Consider the compact group $ G=\operatorname{SO}_3(\mathbb{R}) $. The closed subgroups of $ G $ (other than the trivial group 1 and the whole group $ G $) are exactly $ O_2$, $\operatorname{SO}_2 $ and the finite groups $ C_n$, $D_{2n}$, $T \cong A_4$, $O \cong S_4$, $I \cong A_5 $ (cyclic groups with $ n $ elements, dihedral groups with $ 2n $ elements and the three symmetry groups of the platonic solids). The normalizers of these groups are as follows:

\begin{align*} G&=N_G(G)=N_G(1) \\ O_2&=N_G(O_2)=N_G(\operatorname{SO}_2)=N_G(C_n) \\ I&= N_G(I) \\ O&=N_G(O)=N_G(T)=N_G(D_4) \\ D_{4n} &= N_G(D_{2n}) \end{align*} where in the last equation $ n \geq 3 $. We say a (closed) subgroup is maximal if it is maximal among all proper closed subgroups of $ G $.

Observe that in the example above the maximal subgroups exactly coincide with the self-normalizing subgroups. Namely, $$ O_2, I,O. $$ That the maximal subgroups are all self-normalizing is not too surprising. The normalizer of a closed subgroup is always closed. Thus a maximal subgroup is always either normal or self-normalizing. Since $ G $ is simple, adjoint (i.e. center-free), and connected that means the maximal subgroups must be self-normalizing. However I am a bit surprised that the reverse holds. That is, that every self-normalizing subgroup of $ G $ is maximal. That inspires my question:

For a closed subgroup of a compact Lie group does self-normalizing imply maximal?

This is true for the compact Lie group $ \operatorname{SO}_3(\mathbb{R}) $ and thus also true for $ \operatorname{SU}_2 $. What about the generic case? I am especially interested in $ \operatorname{SU}_3 $.

Answer 1

$\DeclareMathOperator\SO{SO}\DeclareMathOperator\U{U}\DeclareMathOperator\O{O}$Denote by $\U'(n)$ the normalizer of $\U(n)$ in $\mathrm{GL}_{2n}(\mathbf{R})$. It is not hard to see that $\U(n)$ has index 2 in $\U'(n)$, which is generated by $\U(n)$ and by the coordinate-wise complex conjugation. Moreover, $\U'(n)$ is maximal in $\O(2n)$ (if $n\ge 2$).

In $G=\SO(5)$, consider the subgroup $H=\SO(5)\cap (\U'(2)\times \O(1))$ (which contains $\U(2)$ with index 2). I claim that $H$ is self-normalized, but not maximal, in $G=\SO(5)$.

The subgroup $H$ is not maximal in $G$ because it is properly contained in $L=SO(5)\cap (\O(4)\times \O(1))$.

It is self-normalized. Indeed, its action on $\mathbf{R}^5$ has the irreducible decomposition $4\oplus 1$, which is preserved by the normalizer. Hence, the normalizer of $H$ in $G$ equals the normalizer of $H$ inside $\SO(5)\cap (\O(4)\times \O(1))=L$. Using that $\U'(2)$ is maximal in $\O(4)$ one can deduce easily that $H$ is maximal in $L$. Since $H$ is not normal in $L$, it is self-normalized in $L$. Hence $H$ is self-normalized in $G$.