All the automorphisms of $SU(2)$ seem to be inner, which would mean that $\mathrm{Out}$ $SU(2)$ is trivial. Is that correct? Is this true in general $SU(n)$? I can't quite see  any thoughts would be helpful.
3 Answers
$SU(n)$ for $n>2$ has complex fundamental representations. Complex conjugation is an automorphism which exchanges the fundamental representation with its complex conjugate, hence it cannot be an inner automorphism.
Upon further reflection (no pun intended), I think that this is all: basically for simply connected simple Lie groups, the outer automorphisms come from the automorphisms of the Dynkin diagram and for $SU(n)$, $n>2$, the only automorphism is reflection along the midpoint of the diagram. This sends the module with highest weight $(1,0,...,0)$ to $(0,...,0,1)$, hence the fundamental representation to its complex conjugate.

$\begingroup$ Excellent, I can see that now. Thanks very much! And now it seems that complex conjugation sends the fundamental representation of $SU(2)$ to itself. So, am I correct to think that $\mathrm{Out}$ $SU(2)$ is trivial? $\endgroup$ Sep 30, 2010 at 20:06

2$\begingroup$ Yes, I think that's right. The fundamental representation of $SU(2) \cong Sp(1)$ is quaternionic, hence equivalent to its complex conjugate. $\endgroup$ Sep 30, 2010 at 23:57

2$\begingroup$ Comparison with complex Lie algebras and their automorphisms as recalled by algori is the most standard way to understand outer automorphism groups of compact semisimple Lie groups. This makes the answer for SU(n) transparent: the
$A_{n1}$
root system has an outer (= graph) automorphism of order 2 for n>2 but is trivial for n=2 where the Dynkin diagram has just one node. More direct grouptheoretic approaches for classical groups go back a long way (van der Waerden, Dieudonne, many others). All methods require some theory, best done in the uniform language of semisimple Lie theory. $\endgroup$ Oct 3, 2010 at 17:28 
$\begingroup$ Does conjugation descend to an outer automorphism of $ PU_n=SU_n/Z(SU_n) $ as well? Or is every automorphism of $ PU_n $ inner? $\endgroup$ Nov 30, 2022 at 2:22

1$\begingroup$ Ah I see complex conjugation is inner for $ SU_2 $ because it corresponds to conjugation by $ \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} $ $\endgroup$ Feb 21, 2023 at 15:47
Complementing Jose's answer: let $G$ be a complex semisimple simply connected Lie group and let $g$ be the Lie algebra of $G$. The outer automorphism of $g$ is the automorphism group of the Dynkin diagram. Briefly, given an automorphism, we can assume that it preserves a given Cartan subalgebra (or else multiply by an element of the form $Ad(y), g\in G$ that takes one Cartan subalgebra to another one; since the exponential is surjective [edit: no it isn't, as pointed out by Theo, but it is locally] and $Ad(exp(x))=exp(ad(x)),x\in g$, this is an inner automorphism [edit: this only works for $g$ sufficiently close to the unit; in general write $y$ as a product of the exponentials and apply this to each factor]). Any automorphism preserving a Cartan subalgebra $h$ induces an automorphism of the root system, and all automorphisms of the root system arise in this way. Moreover, the automorphisms that induce the identical mapping of the root system are precisely those of the form $exp(ad(x)),x\in h$ (this requires a little check but is not massively difficult).
Now, since complexifying and taking the Lie algebra induces an equivalence of categories of compact simply connected semisimple Lie groups and complex semisimple Lie algebras, the above conclusion holds for the automorphism groups of compact simply connected semisimple Lie groups as well: namely, the outer automorphism group is the automorphism group of the root system (or, which is the same, the automorphism group of the Dynkin diagram).

$\begingroup$ $\exp$ is surjective? In $\operatorname{SL}(2,\mathbb C)$, consider the matrix
$B=\begin{pmatrix}1&1\\0&1\end{pmatrix}$
. Suppose it is $\exp(A)$ for some $A\in\mathfrak{sl}(2,\mathbb C)$. Then the eigenvalues of $A$ are $\pi i + k2\pi i$, and so must be different if they are to sum to $0$. But then $A$ is diagonalizable, but then so is $\exp(A)$, whereas $B$ is not diagonalizable. I do agree that any two Cartan subalgebras are related by an inner automorphism (over $\mathbb C$), but the proof is harder. $\endgroup$ Oct 1, 2010 at 4:16 
$\begingroup$ Theo  my bad. The exponential is surjective for $GL_n(\mathbf{C})$ but not $SL_n$. However it is locally surjective and what one can do instead is write $g$ as a product of the exponentials. Then $Ad(g)$ will be a product of the exponentials of $ad$'s and hence an inner automorphism. $\endgroup$– algoriOct 1, 2010 at 5:06

$\begingroup$ ... and $g$ denotes two different things in the posting. Oh dear.. $\endgroup$– algoriOct 1, 2010 at 5:26

1$\begingroup$ How about nonsimply connected simple real Lie groups ? $\endgroup$ May 4, 2011 at 12:40

$\begingroup$ I'm not entirely sure I believe in that equivalence of categories: aren't there more inner automorphisms of the complex Lie algebra (given by the complex group) than of the compact Lie group? $\endgroup$ Jan 10, 2013 at 12:57
I found the previous answers somewhat unsatisfying. If one takes a complex semisimple Lie group (e.g. $G=GL(n,\mathbb{C})$), then $Out(G)$ is huge, since it is an algebraic group over $\mathbb{Q}$, and therefore acted on by $Aut(\mathbb{C}/\mathbb{Q})$, which is huge. In fact, I don't know what the full automorphism group is. The same applies to $\mathbb{H}^{\ast}$, the nonzero quaternions, when viewed as the CayleyDickson construction applied to $\mathbb{C}$. However, taking the unit quaternions $\mathbb{H}^1\cong SU(2)$ requires complex conjugation, which must preserve $\mathbb{R}$, and therefore gets rid of all of $Aut(\mathbb{C}/\mathbb{Q})$ except for complex conjugation. However, this argument doesn't exclude other outer automorphisms. So I think one can reduce your question to (modulo the previous answers): when does the real form of a Lie group have only continuous (and therefore analytic) automorphisms?

8$\begingroup$ I think this depends on your choice of category. If you consider $Out(G)$ in the category of abstract settheoretic groups, then it is huge, but if you are looking at outer automorphisms of $G$ in the category of Lie groups or topological groups, the question of discontinuous maps is defined away. $\endgroup$– S. Carnahan ♦Oct 1, 2010 at 2:31

1$\begingroup$ Ian  good point! I was implicitly considering the automorphism groups over $\mathbf{C}$, not $\mathbf{Q}$. Complex tori considered as abstract groups have enormous automorphism groups. I'm not sure what the situation is with the simple groups. The Galois trick doesn't work since $Gal(\mathbf{R}/\mathbf{Q})$ is trivial, but there may be other sources of surprises. Maybe this is worthy of a separate MO question? $\endgroup$– algoriOct 1, 2010 at 2:47

3$\begingroup$ I think Scott is right, in that $SU(n)$ usually is the notation for the Lie group (which includes the analytic structure), not just the underlying group. So this is probably what soulphysics has in mind. $\endgroup$– Ian AgolOct 1, 2010 at 2:52

8$\begingroup$ There is an old paper of van der Waerden in which he shows that the topology of a compact semisimple Lie group is determined by the group operation, i.e. any automorphism is automatically continuous, hence smooth. The article (it is in German) can be found here: springerlink.com/content/k613u38120320824/fulltext.pdf $\endgroup$ Oct 1, 2010 at 10:37

$\begingroup$ @Keivan: cool thanks, I didn't know of this paper. $\endgroup$– Ian AgolOct 1, 2010 at 16:14
$D_4$
), but this does not lift to an outer automorphism of $\operatorname{SO}(8)$ or of $\operatorname{O}(8)$. If does lift to $\operatorname{Spin}(8)$. So which group you take matters. $\endgroup$