All the automorphisms of $SU(2)$ seem to be inner, which would mean that $\mathrm{Out}$ $SU(2)$ is trivial. Is that correct? Is this true in general $SU(n)$? I can't quite see  any thoughts would be helpful.
$SU(n)$ for $n>2$ has complex fundamental representations. Complex conjugation is an automorphism which exchanges the fundamental representation with its complex conjugate, hence it cannot be an inner automorphism.
Upon further reflection (no pun intended), I think that this is all: basically for simply connected simple Lie groups, the outer automorphisms come from the automorphisms of the Dynkin diagram and for $SU(n)$, $n>2$, the only automorphism is reflection along the midpoint of the diagram. This sends the module with highest weight $(1,0,...,0)$ to $(0,...,0,1)$, hence the fundamental representation to its complex conjugate.

$\begingroup$ Excellent, I can see that now. Thanks very much! And now it seems that complex conjugation sends the fundamental representation of $SU(2)$ to itself. So, am I correct to think that $\mathrm{Out}$ $SU(2)$ is trivial? $\endgroup$ – soulphysics Sep 30 '10 at 20:06

2$\begingroup$ Yes, I think that's right. The fundamental representation of $SU(2) \cong Sp(1)$ is quaternionic, hence equivalent to its complex conjugate. $\endgroup$ – José FigueroaO'Farrill Sep 30 '10 at 23:57

2$\begingroup$ Comparison with complex Lie algebras and their automorphisms as recalled by algori is the most standard way to understand outer automorphism groups of compact semisimple Lie groups. This makes the answer for SU(n) transparent: the
$A_{n1}$
root system has an outer (= graph) automorphism of order 2 for n>2 but is trivial for n=2 where the Dynkin diagram has just one node. More direct grouptheoretic approaches for classical groups go back a long way (van der Waerden, Dieudonne, many others). All methods require some theory, best done in the uniform language of semisimple Lie theory. $\endgroup$ – Jim Humphreys Oct 3 '10 at 17:28
Complementing Jose's answer: let $G$ be a complex semisimple simply connected Lie group and let $g$ be the Lie algebra of $G$. The outer automorphism of $g$ is the automorphism group of the Dynkin diagram. Briefly, given an automorphism, we can assume that it preserves a given Cartan subalgebra (or else multiply by an element of the form $Ad(y), g\in G$ that takes one Cartan subalgebra to another one; since the exponential is surjective [edit: no it isn't, as pointed out by Theo, but it is locally] and $Ad(exp(x))=exp(ad(x)),x\in g$, this is an inner automorphism [edit: this only works for $g$ sufficiently close to the unit; in general write $y$ as a product of the exponentials and apply this to each factor]). Any automorphism preserving a Cartan subalgebra $h$ induces an automorphism of the root system, and all automorphisms of the root system arise in this way. Moreover, the automorphisms that induce the identical mapping of the root system are precisely those of the form $exp(ad(x)),x\in h$ (this requires a little check but is not massively difficult).
Now, since complexifying and taking the Lie algebra induces an equivalence of categories of compact simply connected semisimple Lie groups and complex semisimple Lie algebras, the above conclusion holds for the automorphism groups of compact simply connected semisimple Lie groups as well: namely, the outer automorphism group is the automorphism group of the root system (or, which is the same, the automorphism group of the Dynkin diagram).

$\begingroup$ $\exp$ is surjective? In $\operatorname{SL}(2,\mathbb C)$, consider the matrix
$B=\begin{pmatrix}1&1\\0&1\end{pmatrix}$
. Suppose it is $\exp(A)$ for some $A\in\mathfrak{sl}(2,\mathbb C)$. Then the eigenvalues of $A$ are $\pi i + k2\pi i$, and so must be different if they are to sum to $0$. But then $A$ is diagonalizable, but then so is $\exp(A)$, whereas $B$ is not diagonalizable. I do agree that any two Cartan subalgebras are related by an inner automorphism (over $\mathbb C$), but the proof is harder. $\endgroup$ – Theo JohnsonFreyd Oct 1 '10 at 4:16 
$\begingroup$ Theo  my bad. The exponential is surjective for $GL_n(\mathbf{C})$ but not $SL_n$. However it is locally surjective and what one can do instead is write $g$ as a product of the exponentials. Then $Ad(g)$ will be a product of the exponentials of $ad$'s and hence an inner automorphism. $\endgroup$ – algori Oct 1 '10 at 5:06

$\begingroup$ ... and $g$ denotes two different things in the posting. Oh dear.. $\endgroup$ – algori Oct 1 '10 at 5:26

1$\begingroup$ How about nonsimply connected simple real Lie groups ? $\endgroup$ – Zoran Skoda May 4 '11 at 12:40

$\begingroup$ I'm not entirely sure I believe in that equivalence of categories: aren't there more inner automorphisms of the complex Lie algebra (given by the complex group) than of the compact Lie group? $\endgroup$ – Michael Thaddeus Jan 10 '13 at 12:57
I found the previous answers somewhat unsatisfying. If one takes a complex semisimple Lie group (e.g. $G=GL(n,\mathbb{C})$), then $Out(G)$ is huge, since it is an algebraic group over $\mathbb{Q}$, and therefore acted on by $Aut(\mathbb{C}/\mathbb{Q})$, which is huge. In fact, I don't know what the full automorphism group is. The same applies to $\mathbb{H}^{\ast}$, the nonzero quaternions, when viewed as the CayleyDickson construction applied to $\mathbb{C}$. However, taking the unit quaternions $\mathbb{H}^1\cong SU(2)$ requires complex conjugation, which must preserve $\mathbb{R}$, and therefore gets rid of all of $Aut(\mathbb{C}/\mathbb{Q})$ except for complex conjugation. However, this argument doesn't exclude other outer automorphisms. So I think one can reduce your question to (modulo the previous answers): when does the real form of a Lie group have only continuous (and therefore analytic) automorphisms?

8$\begingroup$ I think this depends on your choice of category. If you consider $Out(G)$ in the category of abstract settheoretic groups, then it is huge, but if you are looking at outer automorphisms of $G$ in the category of Lie groups or topological groups, the question of discontinuous maps is defined away. $\endgroup$ – S. Carnahan♦ Oct 1 '10 at 2:31

1$\begingroup$ Ian  good point! I was implicitly considering the automorphism groups over $\mathbf{C}$, not $\mathbf{Q}$. Complex tori considered as abstract groups have enormous automorphism groups. I'm not sure what the situation is with the simple groups. The Galois trick doesn't work since $Gal(\mathbf{R}/\mathbf{Q})$ is trivial, but there may be other sources of surprises. Maybe this is worthy of a separate MO question? $\endgroup$ – algori Oct 1 '10 at 2:47

3$\begingroup$ I think Scott is right, in that $SU(n)$ usually is the notation for the Lie group (which includes the analytic structure), not just the underlying group. So this is probably what soulphysics has in mind. $\endgroup$ – Ian Agol Oct 1 '10 at 2:52

8$\begingroup$ There is an old paper of van der Waerden in which he shows that the topology of a compact semisimple Lie group is determined by the group operation, i.e. any automorphism is automatically continuous, hence smooth. The article (it is in German) can be found here: springerlink.com/content/k613u38120320824/fulltext.pdf $\endgroup$ – Keivan Karai Oct 1 '10 at 10:37

$\begingroup$ @Keivan: cool thanks, I didn't know of this paper. $\endgroup$ – Ian Agol Oct 1 '10 at 16:14
$D_4$
), but this does not lift to an outer automorphism of $\operatorname{SO}(8)$ or of $\operatorname{O}(8)$. If does lift to $\operatorname{Spin}(8)$. So which group you take matters. $\endgroup$ – Theo JohnsonFreyd Oct 1 '10 at 4:00