Is the normalizer of a reductive subgroup reductive?

Question

Let $G$ be a reductive algebraic group over an algebraically closed field (of characteristic zero if it matters) and $H \subset G$ a subgroup, also reductive. Is the identity component of the normalizer of $H$ in $G$ always reductive?

Answer 1

I assume $H$ and $G$ are connected, and will explain a purely algebraic proof of the affirmative answer in characteristic 0 (bypassing Cartan involutions) and address the possibility of failure in positive characteristic by putting the question into a broader context that also addresses Jim Humphreys' question of why this is a natural question to wonder about (though I don't know the OP's motivation).

For reasons that will become clearer below, my expectation is that the answer is negative in positive characteristic, due to the failure of nontrivial connected semisimple groups (unlike tori) to have completely reducible representation theory in such characteristics. (EDIT: McNinch has now given such examples over any algebraically closed field of characteristic $p > 0$: the adjoint semisimple subgroup $H = {\rm{PGL}}_n$ inside ${\rm{SL}}({\mathfrak{gl}}_n)$ embedded via "conjugation" for any $n$ divisible by $p$.) No doubt in char. 0 the argument via Cartan involutions by Aakumadula is simpler than what is below in char. 0 (though the proof of Mostow's theorem upon which Aakumadula's argument rests is not easy).

Let $N_G(H)$ and $Z_G(H)$ denote the respective scheme-theoretic normalizer and centralizer of $H$ in $G$, so we want to determine if $N_G(H)^0_{\rm{red}}$ is reductive. The first point is that (in any characteristic) $N_G(H)_{\rm{red}}^0$ is reductive if and only if $Z_G(H)_{\rm{red}}^0$ is reductive. The "only if" direction follows from the normality of $Z_G(H)_{\rm{red}}$ in $N_G(H)_{\rm{red}}$. For the converse, consider the action of $N_G(H)$ on $H$ through conjugation in $G$. This defines a map from $N_G(H)$ to the automorphism functor of $H$ (over $k$), and since $H$ is connected reductive this functor is represented by a smooth $k$-group scheme ${\rm{Aut}}_{H/k}$ whose identity component is the adjoint quotient $H^{\rm{ad}}$ (by SGA3 technology which could be avoided for our purposes by using some auxiliary ad hoc constructions). In particular, the action of $N_G(H)^0$ on $H$ is given by composing a $k$-homomorphism $$f:N_G(H)^0 \rightarrow {\rm{Aut}}_{H/k}^0 = H^{\rm{ad}}$$ with the natural action of $H^{\rm{ad}}$ on $H$. But $f|_H$ is the natural quotient map (due to how $H^{\rm{ad}}$ is identified with a subgroup of ${\rm{Aut}}_{H/k}$) and $(\ker f)_{\rm{red}}^0 = Z_G(H)^0_{\rm{red}}$, so the insensitivity of reductivity to passing to an isogenous quotient and the reductivity of $H^{\rm{ad}}$ imply that $N_G(H)^0_{\rm{red}}$ is reductive if (and only if) $Z_G(H)^0_{\rm{red}}$ is reductive.

So now we are faced with the question of determining when $Z_G(H)^0_{\rm{red}}$ is reductive. Observe that if $H$ were not given as a $k$-subgroup of $G$ (equipped with its conjugation action on $G$) but rather were given to us as an abstract connected reductive group equipped with an abstract action on $G$ as a $k$-group then we could always form the associated connected reductive semidirect product $G' = G \rtimes H$ in which $H$ lies as a subgroup and for which $$Z_{G'}(H) = G^H \times H$$ (where $G^H$ is the scheme-theoretic centralizer in $G$ of the $H$-action). Thus, the reductivity of $(G^H)^0_{\rm{red}}$ is equivalent to that of $Z_{G'}(H)^0_{\rm{red}}$.

To summarize, the original general problem over $k$ of reductivity of the identity component of normalizers of connected reductive subgroups of connected reductive groups is equivalent to the general problem over $k$ of when the underlying reduced scheme of the centralizer $G^H$ for an abstract action of a connected reductive $k$-group $H$ on a connected reductive $k$-group $G$ has reductive identity component. (In particular, the answer is affirmative to one of these two general questions over $k$ if and only if the answer is affirmative to the other general question over $k$.)

We shall now record a purely algebraic result (valid over algebraically closed fields of any characteristic) that gives an affirmative answer to the above questions in characteristic 0 and illuminates the search for counterexamples in positive characteristic. Recall that an affine $k$-group scheme of finite type is called linearly reductive if its finite-dimensional linear representation theory over $k$ is completely reducible. In characteristic 0 this is equivalent to reductivity of the identity component, whereas in positive characteristic the only linearly reductive $k$-group schemes are the affine finite type $k$-groups whose identity component is multiplicative type and component group has order not divisible by char($k$) (by a theorem of Nagata).

$\mathbf{Theorem}$: If $H$ is a linearly reductive $k$-group equipped with an action on a connected reductive $k$-group $G$ then $(G^H)^0_{\rm{red}}$ is reductive.

This (hard) theorem is Proposition A.8.12 of the book "Pseudo-reductive groups". (When ${\rm{char}}(k) = p > 0$ this Theorem is useful for $H = \mu_{p^n}$, not just for the rather classical case when $H$ is a torus.) The proof rests on a beautiful result due independently to Borel (using etale cohomology) and Richardson (using Haboush's theorem), and it also involves considerations with non-representable functors.

Observe that finding a counterexample to the original problem in positive characteristic amounts to showing that the above Theorem has optimal hypotheses; e.g., it fails for some connected semisimple $H$ (and a suitable $G$). This is a very natural question in positive characteristic, and so (as I hope Jim Humphreys will agree) adequately motivates the original problem.

Answer 2

The answer is yes, at least in characteristic zero. There is a Theorem of Mostow which says that $G$ may be viewed as a subgroup of $GL_n$ such that the restriction of the Cartan involution of $GL_n({\mathbb C})$ to $G$ and $H$ gives Cartan involutions on $G$ and $H$, Therefore, the normaliser $N_G(H)$ of $H$ in $G$ is also invariant under this Cartan involution. Hence it is reductive.

Answer 3

In positive characteristic, the normalizer of a (connected) reductive subgroup of a (connected) reductive group is not in general reductive.

I communicated the example below in an emailed answer to a query in April 2002 (it took me some searching in old emails to find it!) At the time I wrote something at the end like "I'm not aware of a good reference" -- this remains true.

Let $k$ be alg. closed of positive characteristic. I will exhibit $H \subset G$ reductive groups such that $C_G(H)$ is not reductive; since $C_G(H)$ is normal in $N_G(H)$, also $N_G(H)$ is not reductive.

Here is the example. Let $n \ge 2$, and consider the adjoint representation $$\operatorname{Ad}: \operatorname{GL}_n \to \operatorname{GL}(\mathfrak{gl}_n)$$ where $\mathfrak{gl}_n$ is the Lie algebra of $\operatorname{GL}_n$.

The image $H \simeq \operatorname{PGL}_n$ of $\operatorname{Ad}$ is a reductive subgroup of $G=\operatorname{GL}(\mathfrak{gl}_n)$, and the centralizer $C_G(H)$ identifies with the group of units of the endomorphism ring $\operatorname{End}_H(\mathfrak{gl}_n)$.

If $(n,p) = 1$, $\mathfrak{gl}_n$ is a semisimple representation of $\operatorname{GL}_n$ with two distinct irreducible factors, so that $\operatorname{End}_H(L) = k \times k$. In this case $C_G(H)$ is a 2 dimensional torus and hence is reductive.

If $(n,p) = p$, $\mathfrak{gl}_n$ is an indecomposable representation with 3 composition factors. Thus the endomorphism ring of $\mathfrak{gl}_n$ is a local ring. It turns out that this endomorphism ring $\operatorname{End}_H(\mathfrak{gl}_n)$ still has dimension 2, but it is now isomorphic to the ring $k[t]/\langle t^2 \rangle$. To exhibit a non-0 nilpotent $H$-endomorphism $t$ of $\mathfrak{gl}_n$, take the following map: to a matrix $X$ in $\mathfrak{gl}_n$, assign the matrix $t(X) = \operatorname{tr}(X).1$ where 1 denotes the $n\times n$ identity matrix, and $\operatorname{tr}()$ denotes the trace. Since $n$ is divisible by $p$, applying $t$ twice gives 0.

The unit group of this local ring is isomorphic to the product of a $1$ dimensional torus and the additive group of the field; such a group is not reductive. (Explicitly: the additive subgroup is precisely the set of all automorphisms $1 + at$ of $\mathfrak{gl}_n$ with $a \in k$.)

More generally, let $H$ be any reductive group, let $V$ be a finite dimensional $H$-module, and write $G = \operatorname{GL}(V)$. Suppose that $V$ is indecomposable, has composition length 3, and that $\operatorname{soc}(V) \simeq V /\operatorname{rad}(V)$. Then $\operatorname{End}_H(V) \simeq k[t] / \langle t^2 \rangle$ and $C_G(H) = \operatorname{End}_H(V)^\times$ is not reductive.

Here are a few more examples of such $H$ and $V$ (I'll write $T(\mu)$ for the indecomposable tilting module with highest weight $\mu$).

• $H = \operatorname{Sp}(W)$, $V = \bigwedge^2 W$ ("exterior square"), when $p>2$ and $\dim W \equiv 0 \pmod p$.

• $H = \operatorname{SO}(W)$, $V = \operatorname{Sym}^2 W$ ("symmetric square"), when $p>2$ and $\dim W \equiv 0 \pmod p$.

• $H = \operatorname{SL}_2$, $V = T(n)$ for $p \le n \le 2p-2$.

• $H = \operatorname{SL}_{\ell + 1}$, $V = T(\varpi_i + \varpi_\ell)$ for $1 \le i < \ell$, when $\ell + 2 - i \equiv 0 \pmod p$, [This more-or-less interpolates the original example since when $i=1$ and $\ell + 1 \equiv 0 \pmod p$, $T(\varpi_1 + \varpi_\ell) \simeq \mathfrak{gl}_{\ell+1}$.]

• $H = \operatorname{SO}_{2\ell}$, $V = T(\varpi_1 + \varpi_\ell)$ when $\ell \equiv 0 \pmod{p}$.

Answer 4

The answer to the question might be yes (over an algebraically closed field of any characteristic), though it's not clearly documented in the literature. First let me add a reference to the theorem of G.D. Mostow used by aakumadula: the paper Self-adjoint groups, Ann. of Math. 62 (1955), 44–55 (MR69830). This is formulated for a "fully reducible" algebraic matrix group $G$ over $\mathbb{R}$ or $\mathbb{C}$, and asserts that a positive definite hermitian form exists relative to which $G$ is self-adjoint. Thus after a change of basis, $G$ is closed under taking complex conjugate transposes. The converse is easy, but the theorem itself relies heavily on the structure theory of relevant Lie groups and would require some work to adapt to arbitrary algebraically closed fields of characteristic 0.

The label reductive applies to the given complex algebraic group $G$ here. By the general definition, a reductive algebraic group has a trivial unipotent radical but need not be connected. In characteristic 0, it follows from Weyl's theorem (indirectly) that a reductive group acts in a completely/fully reducible way in any finite dimensional representation; this is familiar for finite groups. But in prime characteristic this property fails badly for reductive groups including finite groups (except tori), so one has to rely on the full Borel-Chevalley structure theory. This shows that a (say connected) reductive group is an almost-direct product of a torus and a semisimple (connected) algebraic group.

With this in mind, I suspect the answer to the question is yes, though that isn't obvious or well documented in prime characteristic. The crucial case is that of a connected reductive group (call it $H$) which is a subgroup of a reductive group $G$. Write $N$ and $C$ respectively for the normalizer and centralizer of $H$ in $G$. Then $C, H$ are normal subgroups of $N$, while their intersection is central in $H$.

A standard consequence of the isomorphism theory for semisimple groups (Chevalley) is the theorem that inner automorphisms of such a group form a subgroup of finite index in the full automorphism group. This is clearly needed to study $N$ when $H$ is semisimple. In particular, $C \cap H$ is finite and $N$ is the almost-direct product of $C$ and $H$; this makes it most crucial to understand whether $C$ is reductive.

A couple of special cases indicate the problem one still faces in prime characteristic. If $H$ is a torus, an elementary result (rigidity) shows that its centralizer has finite index in its normalizer. From the detailed Borel-Chevalley theory one eventually concludes that the centralizer is reductive (as well as connected). So in this case $N$ is certainly reductive (though perhaps not connected).

On the other hand, suppose $H$ is semisimple of rank 1, such as $\mathrm{SL}_2$. Here the centralizer in $G$ is studied as part of the Bala-Carter parametrization of unipotent classes, but at first under a strong lower bound on the characteristic of the field which for example ensures that the Killing form of the Lie algebra of $G$ (when semisimple) is nondegenerate. Here one finds via the Lie algebra that $C$ is indeed reductive. But smaller primes require much more delicate study.

Nothing here is completely straightforward, so I wonder what the motivation for the question is? This doesn't come up immediately in the basic structure theory.