Let $B$ be the group of upper triangular $n$by$n$ matrices of determinant $1$ over a finite field $F$. What is the group of outer automorphisms of $B$? More generally, what are outer automorphisms of the Borel subgroup of a finite Chevalley group?
2 Answers
All I can suggest is a partial answer. In the narrower case when you look at a Borel subgroup of the finite special linear group, there should be no outer automorphisms arising from conjugation within the general linear group: Here your group (usually denoted something like $B$) lives in a split BNpair for the Chevalley group in question as the subgroup $B$. Like other parabolic subgroups, $B$ is then forced by the axioms to be its own normalizer in the ambient Chevalley group.` Using this standard fact (easily documented in the literature) for the special linear group, it carries over to the general linear group which differs only by scalars. (Similar arguments apply over more general fields.)
The same kind of BNpair structure exists for any finite Chevalley group, along with the same normalizer theorem. But I'm much less certain about what happens when you view the given Borel subgroup just as an abstract solvable group. While its structure as a solvable group is very specialized (as a certain kind of semidirect product), it has usually been studied only in the Chevalley group context.

$\begingroup$ Thanks! Sure $B$ coincides with its normalizer in $GL(n, F)$. But it appears that there are nontrivial outer automorphisms of $B$ , e.g., coming from automorphisms of $F$. Also, in this paper deepblue.lib.umich.edu/handle/2027.42/30473 the group $Out(B)$ is computed when $F$ is the field of $2$ elements. It is surprisingly large (in this case, of course, $B$ is the group of unipotent uppertriangular matrices) $\endgroup$ Commented Mar 5, 2013 at 21:37

1$\begingroup$ @Dmitri: The case
$p=2$
does have a different flavor (with the torus disappering). In general I don't know how to characterize$B$
as an abstract group in a way that helps with the automorphism problem.. Beyond the special linear group, it may turn out that thinking about Borel subgroups or related Lie theory isn't helpful at all. The original notion was tied strongly to the role of this subgroup within a wellbehaved larger group, even over a finite field. $\endgroup$ Commented Mar 6, 2013 at 0:55
Let me add my partial answer. Let us take $B$ to be upper triangular invertible $3\times 3$ matrices over a finite field of $p$ elements, $p\geq 5$ an odd prime. Then I claim that the automorphisms group is generated by the inner ones and the map $\sigma: g\mapsto \kappa ^tg \kappa ^{1}$ where $\kappa$ is the longest Weyl group element in $GL_3$. Let $\theta$ be an automorphism. It takes $U$ to $U$, where $U$ is the unipotent radical of $B$ (since $U$ is the unique $p$sylow subgroup).
After changing $\theta$ an inner conjugation if necessary, we may assume that $T$ goes to $T$ where $T$ is the group of diagonals: The diagonals contain an $l$Sylow subgroup of $B$ where $l$ is a prime dividing $p1$ and we may assume $\theta $ stabilises this $l$Sylow subgroup $H$. Therefore, $\theta $ stabilises the centraliser of this $H$. But the centraliser is $T$.
Since $p\geq 5$ we may take logarithms, and the automorphism $\theta $ takes one root space into another. The root group $X_{13}$ being central, goes into itself. The other two $simple$ root groups $X_{12}$ and $X_{23}$ may be permuted. After changing $\theta $ by $\sigma$ if needed, one can see that $\theta $ stabilises each root group, and after changing $\theta $ by an inner conjugation by an element of $T$, we see that each simple root group is left pointwise fixed by $\theta$. Now by examining the conjugation action of $T$ on these root groups, you can see that $T$ is also pointwise fixed. Since $B$ is generated by $T$ and the simple root groups, it follows that $B$ is also pointwise fixed.
[Remark]: I think this argument extends to $n\times n$ matrices as well , at least when $p>n$ (we took logs). Perhaps with some more group theory tricks, we can avoid logarithms and get this for all $n$ and all odd primes $p$.

$\begingroup$ Thanks, Aakumadula ! Probably one can deduce that $\theta$ (modulo an inner automorphism) respects root subgroups by using that normal abelian subgroups of $B$ are in bijection with positive roots (at least for the type $A_n$). In particular, maximal normal abelian subgroups of $B$ correspond to simple roots. The automorphism $\sigma'$ of $B$ given by composition of taking transpose w.r.t (1n)  (n1) diagonal and taking the inverse. Does this coincide with your $\sigma$? (it corresponds to the symmetry of the $A_n$ Dynkin diagram). $\endgroup$ Commented Mar 6, 2013 at 20:09

1$\begingroup$ Dmitri: you are right. This does give the automorphism $\sigma$. $\endgroup$ Commented Mar 6, 2013 at 22:46