Let $G$ be a reductive algebraic group over an algebraically closed field (of characteristic zero if it matters) and $H \subset G$ a subgroup, also reductive. Is the identity component of the normalizer of $H$ in $G$ always reductive?

1$\begingroup$ Positive characteristic: I don't think anyone spelled this out: for a finite group, the normalizer need not be reductive. EG, for $\mathbb Z/p$ acting by shearing on a 2d vector space, the connected component of the normalizer is $\mathbb G_a$.....This is what xbnv asks for: a nontrivial extension of a rep by itself. But the standard examples of nontrivial extensions of $SL_2$reps have distinct composition factors, so I don't know. $\endgroup$ – Ben Wieland Nov 24 '12 at 20:34

1$\begingroup$ @Ben Wieland: I imposed a connectedness condition in my answer because it seemed almost certain that Martin Orr would have wanted that ("everything" breaks down in the theory of linear algebraic groups when one drops connectedness). One should never say "reductive" without connectedness in positive characteristic, since too many things break down in such cases. $\endgroup$ – user27056 Nov 24 '12 at 22:38
I assume $H$ and $G$ are connected, and will explain a purely algebraic proof of the affirmative answer in characteristic 0 (bypassing Cartan involutions) and address the possibility of failure in positive characteristic by putting the question into a broader context that also addresses Jim Humphreys' question of why this is a natural question to wonder about (though I don't know the OP's motivation).
For reasons that will become clearer below, my expectation is that the answer is negative in positive characteristic, due to the failure of nontrivial connected semisimple groups (unlike tori) to have completely reducible representation theory in such characteristics. (EDIT: McNinch has now given such examples over any algebraically closed field of characteristic $p > 0$: the adjoint semisimple subgroup $H = {\rm{PGL}}_n$ inside ${\rm{SL}}({\mathfrak{gl}}_n)$ embedded via "conjugation" for any $n$ divisible by $p$.) No doubt in char. 0 the argument via Cartan involutions by Aakumadula is simpler than what is below in char. 0 (though the proof of Mostow's theorem upon which Aakumadula's argument rests is not easy).
Let $N_G(H)$ and $Z_G(H)$ denote the respective schemetheoretic normalizer and centralizer of $H$ in $G$, so we want to determine if $N_G(H)^0_{\rm{red}}$ is reductive. The first point is that (in any characteristic) $N_G(H)_{\rm{red}}^0$ is reductive if and only if $Z_G(H)_{\rm{red}}^0$ is reductive. The "only if" direction follows from the normality of $Z_G(H)_{\rm{red}}$ in $N_G(H)_{\rm{red}}$. For the converse, consider the action of $N_G(H)$ on $H$ through conjugation in $G$. This defines a map from $N_G(H)$ to the automorphism functor of $H$ (over $k$), and since $H$ is connected reductive this functor is represented by a smooth $k$group scheme ${\rm{Aut}}_{H/k}$ whose identity component is the adjoint quotient $H^{\rm{ad}}$ (by SGA3 technology which could be avoided for our purposes by using some auxiliary ad hoc constructions). In particular, the action of $N_G(H)^0$ on $H$ is given by composing a $k$homomorphism $$f:N_G(H)^0 \rightarrow {\rm{Aut}}_{H/k}^0 = H^{\rm{ad}}$$ with the natural action of $H^{\rm{ad}}$ on $H$. But $f_H$ is the natural quotient map (due to how $H^{\rm{ad}}$ is identified with a subgroup of ${\rm{Aut}}_{H/k}$) and $(\ker f)_{\rm{red}}^0 = Z_G(H)^0_{\rm{red}}$, so the insensitivity of reductivity to passing to an isogenous quotient and the reductivity of $H^{\rm{ad}}$ imply that $N_G(H)^0_{\rm{red}}$ is reductive if (and only if) $Z_G(H)^0_{\rm{red}}$ is reductive.
So now we are faced with the question of determining when $Z_G(H)^0_{\rm{red}}$ is reductive. Observe that if $H$ were not given as a $k$subgroup of $G$ (equipped with its conjugation action on $G$) but rather were given to us as an abstract connected reductive group equipped with an abstract action on $G$ as a $k$group then we could always form the associated connected reductive semidirect product $G' = G \rtimes H$ in which $H$ lies as a subgroup and for which $$Z_{G'}(H) = G^H \times H$$ (where $G^H$ is the schemetheoretic centralizer in $G$ of the $H$action). Thus, the reductivity of $(G^H)^0_{\rm{red}}$ is equivalent to that of $Z_{G'}(H)^0_{\rm{red}}$.
To summarize, the original general problem over $k$ of reductivity of the identity component of normalizers of connected reductive subgroups of connected reductive groups is equivalent to the general problem over $k$ of when the underlying reduced scheme of the centralizer $G^H$ for an abstract action of a connected reductive $k$group $H$ on a connected reductive $k$group $G$ has reductive identity component. (In particular, the answer is affirmative to one of these two general questions over $k$ if and only if the answer is affirmative to the other general question over $k$.)
We shall now record a purely algebraic result (valid over algebraically closed fields of any characteristic) that gives an affirmative answer to the above questions in characteristic 0 and illuminates the search for counterexamples in positive characteristic. Recall that an affine $k$group scheme of finite type is called linearly reductive if its finitedimensional linear representation theory over $k$ is completely reducible. In characteristic 0 this is equivalent to reductivity of the identity component, whereas in positive characteristic the only linearly reductive $k$group schemes are the affine finite type $k$groups whose identity component is multiplicative type and component group has order not divisible by char($k$) (by a theorem of Nagata).
$\mathbf{Theorem}$: If $H$ is a linearly reductive $k$group equipped with an action on a connected reductive $k$group $G$ then $(G^H)^0_{\rm{red}}$ is reductive.
This (hard) theorem is Proposition A.8.12 of the book "Pseudoreductive groups". (When ${\rm{char}}(k) = p > 0$ this Theorem is useful for $H = \mu_{p^n}$, not just for the rather classical case when $H$ is a torus.) The proof rests on a beautiful result due independently to Borel (using etale cohomology) and Richardson (using Haboush's theorem), and it also involves considerations with nonrepresentable functors.
Observe that finding a counterexample to the original problem in positive characteristic amounts to showing that the above Theorem has optimal hypotheses; e.g., it fails for some connected semisimple $H$ (and a suitable $G$). This is a very natural question in positive characteristic, and so (as I hope Jim Humphreys will agree) adequately motivates the original problem.
The answer is yes, at least in characteristic zero. There is a Theorem of Mostow which says that $G$ may be viewed as a subgroup of $GL_n$ such that the restriction of the Cartan involution of $GL_n({\mathbb C})$ to $G$ and $H$ gives Cartan involutions on $G$ and $H$, Therefore, the normaliser $N_G(H)$ of $H$ in $G$ is also invariant under this Cartan involution. Hence it is reductive.

$\begingroup$ The Cartan involution on $GL_n({\mathbb C})$ is the complex conjugate of the inverse of the transpose. $\endgroup$ – Venkataramana Nov 23 '12 at 15:06


1$\begingroup$ I mean the matrix inverse of the transpose, not the map inverse to the transpose: the involution is the complex conjugate of $(^t A )^{1}$. $\endgroup$ – Venkataramana Nov 24 '12 at 1:06

1$\begingroup$ I consulted Gopal Prasad (University of Michigan). He told me a proof which avoids Mostow's theorem but uses compact groups. Let $K_H$ be maximal compact in $H$ and $K$ a maximal compact in $G$ containing $K_H$, and $\theta $ the Cartan involution on $G$. It is enough to prove $Z_G(H)$ is reductive. Since $H$ is reductive, and $K_H$ is Zariski dense in $H$, it is the same as proving that the centraliser in $G$ of $K_H$ is reductive. But this centraliser s obviously closed under the Cartan involution. $\endgroup$ – Venkataramana Dec 7 '12 at 15:51
In positive characteristic, the normalizer of a (connected) reductive subgroup of a (connected) reductive group is not in general reductive.
I communicated the example below in an emailed answer to a query in April 2002 (it took me some searching in old emails to find it!) At the time I wrote something at the end like "I'm not aware of a good reference"  this remains true.
Let $k$ be alg. closed of positive characteristic. I will exhibit $H \subset G$ reductive groups such that $C_G(H)$ is not reductive; since $C_G(H)$ is normal in $N_G(H)$, also $N_G(H)$ is not reductive.
Here is the example. Let $n \ge 2$, and consider the adjoint representation $$\operatorname{Ad}: \operatorname{GL}_n \to \operatorname{GL}(\mathfrak{gl}_n)$$ where $\mathfrak{gl}_n$ is the Lie algebra of $\operatorname{GL}_n$.
The image $H \simeq \operatorname{PGL}_n$ of $\operatorname{Ad}$ is a reductive subgroup of $G=\operatorname{GL}(\mathfrak{gl}_n)$, and the centralizer $C_G(H)$ identifies with the group of units of the endomorphism ring $\operatorname{End}_H(\mathfrak{gl}_n)$.
If $(n,p) = 1$, $\mathfrak{gl}_n$ is a semisimple representation of $\operatorname{GL}_n$ with two distinct irreducible factors, so that $\operatorname{End}_H(L) = k \times k$. In this case $C_G(H)$ is a 2 dimensional torus and hence is reductive.
If $(n,p) = p$, $\mathfrak{gl}_n$ is an indecomposable representation with 3 composition factors. Thus the endomorphism ring of $\mathfrak{gl}_n$ is a local ring. It turns out that this endomorphism ring $\operatorname{End}_H(\mathfrak{gl}_n)$ still has dimension 2, but it is now isomorphic to the ring $k[t]/\langle t^2 \rangle$. To exhibit a non0 nilpotent $H$endomorphism $t$ of $\mathfrak{gl}_n$, take the following map: to a matrix $X$ in $\mathfrak{gl}_n$, assign the matrix $t(X) = \operatorname{tr}(X).1$ where 1 denotes the $n\times n$ identity matrix, and $\operatorname{tr}()$ denotes the trace. Since $n$ is divisible by $p$, applying $t$ twice gives 0.
The unit group of this local ring is isomorphic to the product of a $1$ dimensional torus and the additive group of the field; such a group is not reductive. (Explicitly: the additive subgroup is precisely the set of all automorphisms $1 + at$ of $\mathfrak{gl}_n$ with $a \in k$.)
More generally, let $H$ be any reductive group, let $V$ be a finite dimensional $H$module, and write $G = \operatorname{GL}(V)$. Suppose that $V$ is indecomposable, has composition length 3, and that $\operatorname{soc}(V) \simeq V /\operatorname{rad}(V)$. Then $\operatorname{End}_H(V) \simeq k[t] / \langle t^2 \rangle$ and $C_G(H) = \operatorname{End}_H(V)^\times$ is not reductive.
Here are a few more examples of such $H$ and $V$ (I'll write $T(\mu)$ for the indecomposable tilting module with highest weight $\mu$).
$H = \operatorname{Sp}(W)$, $V = \bigwedge^2 W$ ("exterior square"), when $p>2$ and $\dim W \equiv 0 \pmod p$.
$H = \operatorname{SO}(W)$, $V = \operatorname{Sym}^2 W$ ("symmetric square"), when $p>2$ and $\dim W \equiv 0 \pmod p$.
$H = \operatorname{SL}_2$, $V = T(n)$ for $p \le n \le 2p2$.
$H = \operatorname{SL}_{\ell + 1}$, $V = T(\varpi_i + \varpi_\ell)$ for $1 \le i < \ell$, when $\ell + 2  i \equiv 0 \pmod p$, [This moreorless interpolates the original example since when $i=1$ and $\ell + 1 \equiv 0 \pmod p$, $T(\varpi_1 + \varpi_\ell) \simeq \mathfrak{gl}_{\ell+1}$.]
$H = \operatorname{SO}_{2\ell}$, $V = T(\varpi_1 + \varpi_\ell)$ when $\ell \equiv 0 \pmod{p}$.

$\begingroup$ @George: This kind of example looks convincing. It would confirm that for a given
$G$
, the normalizer of some connected reductive subgroup$H$
fails to be reductive at least when the prime$p$
fails to be very good for$G$
. I wonder if it's possible that very good primes (say for both$G$
and$H$
) always lead to reductive normalizers? It seems mysterious to have both outcomes possible for different choices of$p$
relative to$G, H$
. More examples and counterexamples might help to decide whether or not there is a systematic pattern. $\endgroup$ – Jim Humphreys Nov 25 '12 at 14:26 
2$\begingroup$ @Jim: I think I'm skeptical that "very good primes" are the issue. If I view $M=\operatorname{GL}(L)$ as a Levi factor of a parabolic subgroup of $G=\operatorname{GL}(L \oplus k)$ and take the subgroup $H$ of $M$ as in the construction of my answer, I believe that $C_G(H)$ is still not reductive. $\endgroup$ – George McNinch Nov 25 '12 at 14:39

1$\begingroup$ Here's a slightly different construction: If $V$ is a representation with nonsplit quotient $W$, then $V\oplus W$ admits an equivariant shear from $W$ to $V$ that cannot be complemented without splitting $V$. So the centralizer of the group in $GL(V\oplus W)$ is not reductive, so the normalizer is not. $\endgroup$ – Ben Wieland Nov 26 '12 at 16:19
The answer to the question might be yes (over an algebraically closed field of any characteristic), though it's not clearly documented in the literature. First let me add a reference to the theorem of G.D. Mostow used by aakumadula: the paper Selfadjoint groups, Ann. of Math. 62 (1955), 44–55 (MR69830). This is formulated for a "fully reducible" algebraic matrix group $G$ over $\mathbb{R}$ or $\mathbb{C}$, and asserts that a positive definite hermitian form exists relative to which $G$ is selfadjoint. Thus after a change of basis, $G$ is closed under taking complex conjugate transposes. The converse is easy, but the theorem itself relies heavily on the structure theory of relevant Lie groups and would require some work to adapt to arbitrary algebraically closed fields of characteristic 0.
The label reductive applies to the given complex algebraic group $G$ here. By the general definition, a reductive algebraic group has a trivial unipotent radical but need not be connected. In characteristic 0, it follows from Weyl's theorem (indirectly) that a reductive group acts in a completely/fully reducible way in any finite dimensional representation; this is familiar for finite groups. But in prime characteristic this property fails badly for reductive groups including finite groups (except tori), so one has to rely on the full BorelChevalley structure theory. This shows that a (say connected) reductive group is an almostdirect product of a torus and a semisimple (connected) algebraic group.
With this in mind, I suspect the answer to the question is yes, though that isn't obvious or well documented in prime characteristic. The crucial case is that of a connected reductive group (call it $H$) which is a subgroup of a reductive group $G$. Write $N$ and $C$ respectively for the normalizer and centralizer of $H$ in $G$. Then $C, H$ are normal subgroups of $N$, while their intersection is central in $H$.
A standard consequence of the isomorphism theory for semisimple groups (Chevalley) is the theorem that inner automorphisms of such a group form a subgroup of finite index in the full automorphism group. This is clearly needed to study $N$ when $H$ is semisimple. In particular, $C \cap H$ is finite and $N$ is the almostdirect product of $C$ and $H$; this makes it most crucial to understand whether $C$ is reductive.
A couple of special cases indicate the problem one still faces in prime characteristic. If $H$ is a torus, an elementary result (rigidity) shows that its centralizer has finite index in its normalizer. From the detailed BorelChevalley theory one eventually concludes that the centralizer is reductive (as well as connected). So in this case $N$ is certainly reductive (though perhaps not connected).
On the other hand, suppose $H$ is semisimple of rank 1, such as $\mathrm{SL}_2$. Here the centralizer in $G$ is studied as part of the BalaCarter parametrization of unipotent classes, but at first under a strong lower bound on the characteristic of the field which for example ensures that the Killing form of the Lie algebra of $G$ (when semisimple) is nondegenerate. Here one finds via the Lie algebra that $C$ is indeed reductive. But smaller primes require much more delicate study.
Nothing here is completely straightforward, so I wonder what the motivation for the question is? This doesn't come up immediately in the basic structure theory.

$\begingroup$ I confess that the question had no great motivation at all. I was considering the conjugacy class of $H$ in $G$, and hence the normalizer comes up. I noticed that when I picked examples, the normalizer is always reductive, but I couldn't prove that this was true in general. $\endgroup$ – Martin Orr Nov 24 '12 at 16:15

$\begingroup$ @Martin: The question is natural but doesn't seem to come up in the standard structure theory. Aside from motivation, it makes a big difference whether you are interested only in the complex field or want an arbitrary algebraically closed field of char 0: the methods may differ. I still hope that a fairly "elementary" algebraic proof (based on Bourbaki's Chapter 1) will be found here, by translating to the Lie algebra setting. In char
$p>0$
there might be counterexamples for small primes, even though the rank 1 case I outlined is encouraging for$p\gg 0$
. $\endgroup$ – Jim Humphreys Nov 24 '12 at 21:46 
1$\begingroup$ @Jim: Since $G,H$ and the normaliser are all defined and split over a finitely generated field $k$ over the prime field, the question of "reductivity" does not change over extensions of the field $k$. We may thus assume that in characteristic zero, the field $k$ is embedded in ${\mathbb C}$ and then argue using Mostow's theorem. $\endgroup$ – Venkataramana Nov 25 '12 at 3:59

1$\begingroup$ @Aakumadula: That seams reasonable, though I still wonder whether a more elementary proof in characteristic 0 (say in the equivalent Lie algebra setting) might exist. Mostow's theorem relies heavily on Lie group theory even though the desired result is purely algebraic. (Though Mostow is an old acquaintance of mine.) $\endgroup$ – Jim Humphreys Nov 25 '12 at 14:29

$\begingroup$ Concerning the rank 1 situation: given (say) a $p$nilpotent element $X$ of $\mathfrak{g}$, if the char is "very good" for $G$, there is indeed a rank 1 subgroup $H$ of $G$ with $X \in \operatorname{Lie}(H)$ for which $C_G(H)$ is reductive. But not every rank 1 subgroup has reductive centralizer, even when $p$ is "not too tiny". e.g. if $H = \operatorname{SL}_2$ and $V$ is the $H$representation $V = T(p)$ where $T(p)$ is the tilting module of highest weight $p$ (say) the argument sketched in my answer shows that $C_G(H)$ is not reductive, where $G=\operatorname{GL}(V)$. $\endgroup$ – George McNinch Nov 25 '12 at 21:12