Can solvable connected Lie groups have maximal subgroups?

Question

Cross-posted from MSE.

Many interesting manifolds can be expressed as $ G/H $ for $ G $ a connected Lie group and $ H $ a maximal closed subgroup. Examples include the projective spaces $ \mathbb{C}P^n \cong \operatorname{SU}_n/U_{n-1} $ where $ U_{n-1} $ is maximal for $ n \geq 3 $, and $ \mathbb{R}P^n \cong \operatorname{SO}_n/O_{n-1} $, again $ O_{n-1} $ is maximal for $ n \geq 3 $. Another example is the Poincare homology sphere $ \operatorname{SO}_3(\mathbb{R})/A_5 $.

Solvmanifolds provide many interesting examples of manifolds, especially of torus bundles over tori (a solvmanifold is a manifold of the form $ G/H $ for $ G $ a solvable Lie group).

The examples I list above of manifolds $ G/H $, $ H $ maximal, all have the property that $ G $ is connected semisimple (indeed simple).

This leads me to wonder about the opposite case: maximal closed subgroups $ H $ of connected solvable Lie groups $ G $. Do they even exist?

Let $ G $ be a connected Lie group.

If $ G $ is abelian then certainly $ G $ does not have any maximal closed subgroups. Does the same hold for $ G $ solvable?

$\DeclareMathOperator\Ab{Ab}$Comment: Let $ G' $ be the commutator subgroup of the connected group $ G $. Let $$ \Ab: G \to G/G' $$
be the abelianization map. If $ H $ is a maximal closed subgroup of $ G $ then we must have $$ \Ab(H)=G/G' $$ because if $ \Ab(H) $ was properly contained then $ \Ab(H) $ would be a maximal closed subgroup of the connected abelian group $ G/G' $ which is impossible. In particular that implies that $ H $ does not contain $ G' $ (because if $ \Ab(H)=G/G' $ and $ H $ contained $ G' $ that would imply that $ H $ is all of $ G $, contradicting maximality).

Update: Recall that $ G $ is always a connected Lie group.

If $ G $ is nilpotent then there does not exist any maximal proper closed subgroup (proved in the original answer of LSpice).

If $ G $ is non-nilpotent then there does exist some maximal proper closed subgroup. We prove this with two cases.

$\DeclareMathOperator\Lie{Lie}$If the non-nilpotent group $ G $ is moreover non-solvable then we appeal to basically a Levi decomposition. $ \Lie(G) $ can be written as $$ \Lie(G)= \mathfrak{g}_\text{solv} \rtimes \mathfrak{g}_\text{ss}. $$ Let $ G_\text{solv} $ be a maximal solvable closed connected subgroup of $ G $ corresponding to the Lie subalgebra $ \mathfrak{g}_\text{solv} $. Let $ G_\text{ss} $ be a maximal semisimple closed connected subgroup of $ G $, corresponding to the Lie subalgebra $ \mathfrak{g}_\text{ss} $. Pick $ H_\text{max} $ to be a maximal proper closed subgroup of $ G_\text{ss} $ (there are lots of fairly well known maximal closed subgroups of semisimple groups). Then the group generated by $ G_\text{solv} $ and $ H_\text{max} $ should be roughly $ G_\text{solv} \rtimes H_\text{max} $ and should be a maximal proper closed subgroup of $ G $.

For the case that the non-nilpotent group $ G $ is solvable, then apply the answer from YCor (accepted below) which shows that a solvable non-nilpotent Lie group $ G $ must have a quotient which is one of the four solvable non-nilpotent subgroups of $$ \operatorname{AGL}_1(\mathbb{C}) \cong \mathbb{C}^* \ltimes \mathbb{C} $$ that YCor lists below. In that case there is a maximal proper closed subgroup of the quotient so we can pullback through the quotient map to get a maximal proper closed subgroup of the solvable non-nilpotent Lie group $ G $.

This proves the claim, from YCor's comment, that a connected Lie group $ G $ has a maximal proper closed subgroup if and only if $ G $ is non-nilpotent.

Answer 1

$\newcommand{\g}{\mathfrak{g}}$We can obtain the characterization by classifying just non-nilpotent connected Lie groups.

Let $G$ be a solvable, non-nilpotent connected Lie group. Suppose that every quotient Lie group of $G$ of dimension $<\dim(G)$ is nilpotent. Then $G$ has the form $H\ltimes V$ with both $H,V$ abelian of dimension $\le 2$ and $H$ acting nontrivially irreducibly on $V$ (see below for a list of possibilities).

First suppose that $G$ has a trivial center.

Let $\g$ be its Lie algebra, $(\g^i)_{i\ge 1}$ the lower central series and $\g^\infty=\bigcap_{i\ge 1}\g^i$, and $(G^i)$ the corresponding subgroups (a priori not closed). Note that $G^\infty$ is nilpotent (being contained in the derived subgroup $G^2$). Consider the adjoint representation of $G$. Since we can triangulate it (after complexification), we see that $\g^\infty$ maps to upper triangular matrices with zero diagonal. Hence $G^\infty$ maps to upper unipotent matrices. Thus $G^\infty$ is closed in $G$, and is simply connected. The action of $G$ on $G^\infty/[G^\infty,G^\infty]$ is not unipotent, since otherwise the action on $G^\infty$ would also be, and we would deduce that $G$ is nilpotent. So $G/[G^\infty,G^\infty]$ is a non-nilpotent Lie quotient of $G$, and hence, by the assumption, we deduce that $G^\infty$ is abelian. We claim that $\g^\infty$ is an irreducible module over $G/G^\infty$. Indeed, if it has a nonzero submodule, this corresponds to a connected nontrivial normal subgroup $N$ of $G$, contained in $G^\infty$. By the initial assumption, $G/N$ is nilpotent. But $G/G^\infty$ is the largest nilpotent quotient of $G$. So $N=G$. This proves irreducibility. Thus $G^\infty$ has dimension 1 or 2.

Let $\mathfrak{h}$ be a Cartan subalgebra in $\g$ (in the sense of Bourbaki: self-normalizing nilpotent subalgebra), and $H$ the corresponding subgroup (a priori possibly not closed). Then $\g=\mathfrak{h}+\g^\infty$. The intersection $\mathfrak{h}\cap\g^\infty$ is normalized by $\mathfrak{h}$, hence by $\mathfrak{g}$ (since $\g^\infty$ is abelian). By irreducibility, it is equal to either $\{0\}$ or $\g^\infty$. The latter is impossible, since $\mathfrak{h}$ is nilpotent. So $\g=\mathfrak{h}\ltimes\g^\infty$. By irreducibility, $[\mathfrak{h},\mathfrak{h}]$ centralizes $\g^\infty$. Let $\mathfrak{z}$ be the center of $\mathfrak{h}$. Then $[\mathfrak{h},\mathfrak{h}]\cap\mathfrak{z}$ is central in $\mathfrak{g}$, hence is zero. Since every nonzero ideal of a nilpotent Lie algebra meets the center, we deduce that $[\mathfrak{h},\mathfrak{h}]=0$: so $\mathfrak{h}$ is abelian. Also, the centralizer of $\g^\infty$ in $\mathfrak{h}$ is trivial, for the same reason.

Finally, we have one of the following

1. $\dim(\g^\infty)=1$: then $\g=\mathbf{R}\ltimes\mathbf{R}$, so $G$ is the non-abelian $\mathbf{R}_{>0}\ltimes\mathbf{R}$.
2. $\dim(\g^\infty)=2$, $\dim(\mathfrak{h})=1$. Then $G=\mathrm{U}(1)\ltimes\mathbf{C}$, or $G=\mathbf{R}\ltimes\mathbf{C}$ (acting by an unbounded 1-parameter subgroup of $\mathbf{C}^*$)
3. $\dim(\g^\infty)=2$, $\dim(\mathfrak{h})=2$: then $G=\mathbf{C}^*\ltimes\mathbf{C}$.

If the center of $G$ is nontrivial, then it is discrete and the quotient has a trivial center. We obtain the possible connected covers of the previous examples (i.e., of $\mathrm{U}(1)\ltimes\mathbf{C}$ and of $\mathbf{C}^*\ltimes\mathbf{C}$).

(Note that $H$ is eventually closed: we see this by first viewing the cover $H\ltimes G^\infty$ of $G$, and then observing that this cover is trivial.)

---

Turning back to the issue of maximal subgroups: in all cases we have obtained a group of the form $H\ltimes V$ where $H$ acts irreducibly on $V$. So $H$ is a maximal proper subgroup, which is closed.

If $G$ is an arbitrary non-nilpotent connected Lie group, then it has a quotient of this form, and hence admits a maximal proper subgroup that is closed.

Answer 2

A connected, soluble nilpotent Lie group $G$ has no maximal proper, closed subgroup. (Thanks to @YCor for pointing out, with a counterexample, that my original claim was incorrect.)

We proceed by induction on the dimension of $G$. If the dimension is $0$, then $G$ is trivial, and we are done; so suppose that the dimension is positive, and hence that $\operatorname Z(G)^\circ$ is a positive-dimensional subgroup of $G$.

Suppose that $H$ is a maximal proper, closed subgroup of $G$. If $H$ contains $\operatorname Z(G)^\circ$, then $H/{\operatorname Z(G)}^\circ$ is a maximal proper, closed subgroup of $G/{\operatorname Z(G)}^\circ$, which is a contradiction. Therefore, $H$ is a proper subgroup of $H\cdot\operatorname Z(G)^\circ$. By maximality, we have that $H\cdot\operatorname Z(G)^\circ$ is dense in $G$. (It seems plausible that $H\cdot\operatorname Z(G)^\circ$ is already closed, but I do not know how to prove it.) Thus, for every pair of elements $g, g' \in G$, we have sequences $((h_n, z_n))_n$ and $((h_n', z_n'))_n$ in $H \times \operatorname Z(G)^\circ$ whose images under the multiplication map $H \times \operatorname Z(G)^\circ \to G$ converge to $g$ and $g'$. Then the sequence $([h_n z_n, h_n'z_n^{\prime\,{-1}}])_n$ equals $([h_n, h_n'])_n$, hence lies in $H$; and converges to $[g, g']$, which therefore belongs to $H$. Then $H$ contains the derived subgroup of $G$, which, as you have observed, is a contradiction.

Answer 3

The following is a complement to my other answer (which I thought was about all soluble groups but @YCor pointed out was only about nilpotent groups), addressing a special case of @YCor's conjecture. I am used to thinking of algebraic groups, so I have restricted to linear Lie groups. I hope I have not made any mistakes translating from the algebraic-group setting. Updated later after @YCor's answer: I think our approaches follow the same basic outline, but theirs handles general soluble groups by using more knowledge of their structure theory.

$\DeclareMathOperator\Lie{Lie}$A linear connected, soluble Lie group $G$—in case the definition is not standard, I mean a group of the form $G = \mathbf G(\mathbb R)^\circ$, where $\mathbf G$ is a connected, soluble algebraic group over $\mathbb R$—admits a decomposition $G = T \ltimes U$, where $T$ is of the form $(S^1)^m \times (\mathbb R^\times_{> 0})^n$ for some $m$ and $n$, $U$ is unipotent, and $\Lie(U)$ is a semisimple $T$-module [Bo, Theorem 10.6(4)]. (@YCor points out in a comment that this is probably more restrictive than the general definition of ‘linear Lie group’. Maybe something like ‘algebraic Lie group’ is more common?—although my passage to the identity component will often mean that I am not actually dealing with the group of rational points of an algebraic group.)

If $T$ acts trivially on $U$, then $G$ is nilpotent, and then my other answer shows that $G$ has no maximal proper, closed subgroups.

Suppose that $T$ does not act trivially on $U$, so that (since $U^T$ is connected) $\Lie(U)^T = \Lie(U^T)$ is a proper subspace of $\Lie(U)$. Let $(U_i)_{i = 1}^\infty$ be the descending central series of $U$. Since $\Lie(U)$ is $T$-equivariantly isomorphic to $\bigoplus_{i = 1}^\infty \Lie(U_i/U_{i + 1})$, there is some index $i$ such that $T$ acts non-trivially on $\Lie(U_i/U_{i + 1})$, and hence on $U_i/U_{i + 1}$. Let $n$ be the least such index. If $n \ne 1$, then the commutator map $\Lie(U_1/U_2) \otimes \Lie(U_{n - 1}/U_n) \to \Lie(U_n/U_{n + 1})$ is a $T$-equivariant surjection from a $T$-isotrivial module, so must have $T$-isotrivial image, which is a contradiction. Therefore, $n$ equals $1$, so $T$ does not act trivially on $U_1/U_2$. By [BS, Corollary 9.9], there is a $T$-stable complement $\overline V$ to $(U_1/U_2)^T$ in $U_1/U_2$ that carries a $T$-equivariant vector-group structure. Let $\smash{\overline V}'$ be a maximal proper sub-$T$-module of $\overline V$, and write $V'$ for the pullback of $\smash{\overline V}'\cdot(U_1/U_2)^T$ to $U$. Then $T \ltimes V'$ is a maximal proper subgroup of $G$, and it is closed.

[BS] Borel, A.; Springer, T. A. Rationality properties of linear algebraic groups. II. Tohoku Math. J. (2) 20 (1968), 443–497. (Reviewer: R. Steinberg) 14.50

[Bo] Borel, Armand. Linear algebraic groups. Second edition. Graduate Texts in Mathematics, 126. Springer-Verlag, New York, 1991. xii+288 pp. ISBN: 0-387-97370-2 (Reviewer: F. D. Veldkamp) 20-01 (20Gxx)