In positive characteristic, the normalizer of a (connected) reductive subgroup of a (connected) reductive group is not in general reductive.

I communicated the example below in an emailed answer to a query in April 2002 (it took me some searching in old emails to find it!) At the time I wrote something at the end like "I'm not aware of a good reference" -- this remains true.

Let $k$ be alg. closed of positive characteristic. I will exhibit $H < G$ two reductive groups such that $C_G(H)$ is not reductive; since $C_G(H)$ is normal in $N_G(H)$, also $N_G(H)$ is not reductive.

A slight modification of the example also shows that the group $C_G(H)/Z(H)$ can be unipotent (here $Z(H)$ is the center of $H$).

Here is the example. Let $n \ge 2$, and consider the adjoint representation $$\operatorname{Ad}: \operatorname{GL}(n,k) \to \operatorname{GL}(L)$$ where $L$ is the Lie algebra of $\operatorname{GL}(n,k)$.

The image $H$ of $\operatorname{Ad}$ is a reductive subgroup of $G=\operatorname{GL}(L)$, and the centralizer $C_G(H)$ identifies with the group of units of the endomorphism ring $\operatorname{End}_H(L)$.

If $(n,p) = 1$, $L$ is a semisimple representation of $\operatorname{GL}(n,k)$ ith two distinct irreducible factors, so $\operatorname{End}_H(L) = k \times k$, and in this case $C_G(H)$ is therefore a 2 dimensional torus hence is reductive.

If $(n,p) = p$, $L$ is an indecomposable representation with 3 composition factors. Thus the endomorphism ring of $L$ is a local ring. It turns out that this endomorphism ring $\operatorname{End}_H(L)$ still has dimension 2, but it is now isomorphic to the ring $k[t]/(t^2)$. To exhibit a non-0 nilpotent $H$-endomorphism $t$ of $L$, take the following map: to a matrix $X$ in $L$, assign the matrix  $t(X) = \operatorname{tr}(X).1$ where 1 denotes the $n\times n$ identity matrix, and $\operatorname{tr}()$ denotes the trace. Since  $n$ is divisible by $p$, applying $t$ twice gives 0.

The unit group of this local ring is isomorphic to the product of a $1$ dimensional torus and the additive group of the field; such a group is not reductive. The additive subgroup is precisely the set of all automorphisms $1 + at$ of $L$ with $a \in k$.

Now restrict $\operatorname{Ad}$ to $\operatorname{SL}(n,k)$; this is not the adjoint representation of $\operatorname{SL}(n,k)$ as we do not change $L$. Let $H'$ be the image of $\operatorname{SL}(n,k)$ in $G'=\operatorname{SL}(L)$. Then $C_{G'}(H')$ is isomorphic to the additive group of the field and is thus unipotent.

In positive characteristic, the normalizer of a (connected) reductive subgroup of a (connected) reductive group is not in general reductive.

I communicated the example below in an emailed answer to a query in April 2002 (it took me some searching in old emails to find it!) At the time I wrote something at the end like "I'm not aware of a good reference" -- this remains true.

Let $k$ be alg. closed of positive characteristic. I will exhibit $H \subset G$ reductive groups such that $C_G(H)$ is not reductive; since $C_G(H)$ is normal in $N_G(H)$, also $N_G(H)$ is not reductive.

Here is the example. Let $n \ge 2$, and consider the adjoint representation $$\operatorname{Ad}: \operatorname{GL}_n \to \operatorname{GL}(\mathfrak{gl}_n)$$ where $\mathfrak{gl}_n$ is the Lie algebra of $\operatorname{GL}_n$.

The image $H \simeq \operatorname{PGL}_n$ of $\operatorname{Ad}$ is a reductive subgroup of $G=\operatorname{GL}(\mathfrak{gl}_n)$, and the centralizer $C_G(H)$ identifies with the group of units of the endomorphism ring $\operatorname{End}_H(\mathfrak{gl}_n)$.

If $(n,p) = 1$, $\mathfrak{gl}_n$ is a semisimple representation of $\operatorname{GL}_n$ with two distinct irreducible factors, so that $\operatorname{End}_H(L) = k \times k$. In this case $C_G(H)$ is a 2 dimensional torus and hence is reductive.

If $(n,p) = p$, $\mathfrak{gl}_n$ is an indecomposable representation with 3 composition factors. Thus the endomorphism ring of $\mathfrak{gl}_n$ is a local ring. It turns out that this endomorphism ring $\operatorname{End}_H(\mathfrak{gl}_n)$ still has dimension 2, but it is now isomorphic to the ring $k[t]/\langle t^2 \rangle$. To exhibit a non-0 nilpotent $H$-endomorphism $t$ of $\mathfrak{gl}_n$, take the following map: to a matrix $X$ in $\mathfrak{gl}_n$, assign the matrix  $t(X) = \operatorname{tr}(X).1$ where 1 denotes the $n\times n$ identity matrix, and $\operatorname{tr}()$ denotes the trace. Since  $n$ is divisible by $p$, applying $t$ twice gives 0.

The unit group of this local ring is isomorphic to the product of a $1$ dimensional torus and the additive group of the field; such a group is not reductive. (Explicitly: the additive subgroup is precisely the set of all automorphisms $1 + at$ of $\mathfrak{gl}_n$ with $a \in k$.)

More generally, let $H$ be any reductive group, let $V$ be a finite dimensional $H$-module, and write $G = \operatorname{GL}(V)$. Suppose that $V$ is indecomposable, has composition length 3, and that $\operatorname{soc}(V) \simeq V /\operatorname{rad}(V)$. Then $\operatorname{End}_H(V) \simeq k[t] / \langle t^2 \rangle$ and $C_G(H) = \operatorname{End}_H(V)^\times$ is not reductive.

Here are a few more examples of such $H$ and $V$ (I'll write $T(\mu)$ for the indecomposable tilting module with highest weight $\mu$).

• $H = \operatorname{Sp}(W)$, $V = \bigwedge^2 W$ ("exterior square"), when $p>2$ and $\dim W \equiv 0 \pmod p$.

• $H = \operatorname{SO}(W)$, $V = \operatorname{Sym}^2 W$ ("symmetric square"), when $p>2$ and $\dim W \equiv 0 \pmod p$.

• $H = \operatorname{SL}_2$, $V = T(n)$ for $p \le n \le 2p-2$.

• $H = \operatorname{SL}_{\ell + 1}$, $V = T(\varpi_i + \varpi_\ell)$ for $1 \le i < \ell$, when $\ell + 2 - i \equiv 0 \pmod p$, [This more-or-less interpolates the original example since when $i=1$ and $\ell + 1 \equiv 0 \pmod p$, $T(\varpi_1 + \varpi_\ell) \simeq \mathfrak{gl}_{\ell+1}$.]

• $H = \operatorname{SO}_{2\ell}$, $V = T(\varpi_1 + \varpi_\ell)$ when $\ell \equiv 0 \pmod{p}$.