Automorphism group of real orthogonal Lie groups

Question

I would like to understand what is the "outer-automorphism group" $Out$ of $SO(p,q)$ and $O(p,q)$, where $p+q >0$ and $pq \neq 0$. My working definition of $Out$ is as follows:

Let us denote by $Aut(G)$ the automorphism group of a Lie group $G$. I take the inner-automorphism group $Inn(G)$ of $G$ to be all elements $K\in Aut(G)$ for which there exists a $g\in G$ such that $K = Ad_{g}$, namely $K(h) = g h g^{-1}$ for all $h\in G$. $Inn(G)$ is a normal subgroup of $Aut(G)$ and then $Out(G) = Aut(G)/Inn(G)$ is a group which I define to be the outer-morphism group of $G$. I have not been able to find what $Out(G)$ is for $G = SO(p,q), O(p,q)$.

I have noticed that there are many references dealing with the outer-automorphism group of complex Lie algebras, which can be read off from their Dynkin diagram. However, $\mathfrak{so}(p,q)\simeq\mathfrak{o}(p,q)$ is not a complex Lie algebra but a real form. I don't know how the outer-automorphism group of a simple real Lie algebra can be computed in general. In fact, Wikipedia says that the characterization of the outer-automorphism group of a real simple Lie algebra in terms of a short exact sequence involving the full and inner autmorphisms groups (a result classical for complex Lie algebras) was only obtained as recently as in 2010! In any case, I expect the answer to my question to be even more involved since I am not interested in the outer-automorphism group of a real Lie algebra but of the full real Lie group, in my case $SO(p,q)$ and $O(p,q)$. If I am not mistaken, for $q=0$ and $p = even$ we have $O(p,0) = SO(p,0)\rtimes\mathbb{Z}_{2}$, where $\mathbb{Z}_{2}$ is the outer-automorphism group of $SO(p,0)$, so $Out(SO(p,0)) = \mathbb{Z}_{2}$.

Thanks.

Answer 1

Let's first address your comment in response to Igor Rivin's answer: why don't we find this topic addressed in textbooks on Lie groups? Beyond the definite (= compact) case, disconnectedness issues become more complicated and your question is thereby very much informed by the theory of linear algebraic groups $G$ over $\mathbf{R}$. That in turn involves two subtle aspects (see below) that are not easy to express solely in analytic terms and are therefore beyond the level of such books (which usually don't assume familiarity with algebraic geometry at the level needed to work with linear algebraic groups over a field such as $\mathbf{R}$ that is not algebraically closed). And books on linear algebraic groups tend to say little about Lie groups.

The first subtlety is that $G(\mathbf{R})^0$ can be smaller than $G^0(\mathbf{R})$ (i.e., connectedness for the analytic topology may be finer than for the Zariski topology), as we know already for indefinite orthogonal groups, and textbooks on Lie groups tend to focus on the connected case for structural theorems. It is a deep theorem of Elie Cartan that if a linear algebraic group $G$ over $\mathbf{R}$ is Zariski-connected semisimple and simply connected (in the sense of algebraic groups; e.g., ${\rm{SL}}_n$ and ${\rm{Sp}}_{2n}$ but not ${\rm{SO}}_n$) then $G(\mathbf{R})$ is connected, but that lies beyond the level of most textbooks. (Cartan expressed his result in analytic terms via anti-holomorphic involutions of complex semisimple Lie groups, since there was no robust theory of linear algebraic groups at that time.) The group $G(\mathbf{R})$ has finitely many connected components, but that is not elementary (especially if one assumes no knowledge of algebraic geometry), and the theorem on maximal compact subgroups of Lie groups $H$ in case $\pi_0(H)$ is finite but possibly not trivial appears to be treated in only one textbook (Hochschild's "Structure of Lie groups", which however does not address the structure of automorphism groups); e.g., Bourbaki's treatise on Lie groups assumes connectedness for much of its discussion of the structure of compact Lie groups.

The second subtlety is that when the purely analytic operation of "complexification" for Lie groups (developed in Hochschild's book too) is applied to the Lie group of $\mathbf{R}$-points of a (Zariski-connected) semisimple linear algebraic group, it doesn't generally "match" the easier algebro-geometric scalar extension operation on the given linear algebraic group (e.g., the complexification of the Lie group ${\rm{PGL}}_3(\mathbf{R})$ is ${\rm{SL}}_3(\mathbf{C})$, not ${\rm{PGL}}_3(\mathbf{C})$). Here too, things are better-behaved in the "simply connected" case, but that lies beyond the level of introductory textbooks on Lie groups.

---

Now let us turn to your question. Let $n = p+q$, and assume $n \ge 3$ (so the Lie algebra is semisimple; the cases $n \le 2$ can be analyzed directly anyway). I will only address ${\rm{SO}}(p,q)$ rather than ${\rm{O}}(p, q)$, since it is already enough of a headache to keep track of disconnected effects in the special orthogonal case. To be consistent with your notation, we'll write $\mathbf{O}(p,q) \subset {\rm{GL}}_n$ to denote the linear algebraic group over $\mathbf{R}$ "associated" to the standard quadratic form of signature $(p, q)$ (so its group of $\mathbf{R}$-points is what you have denoted as ${\rm{O}}(p,q)$), and likewise for ${\mathbf{SO}}(p,q)$.

We will show that ${\rm{SO}}(p, q)$ has only inner automorphisms for odd $n$, and only the expected outer automorphism group of order 2 (arising from reflection in any nonzero vector) for even $n$ in both the definite case and the case when $p$ and $q$ are each odd. I will leave it to someone else to figure out (or find a reference on?) the case with $p$ and $q$ both even and positive.

We begin with some preliminary comments concerning the definite (= compact) case for all $n \ge 3$, for which the Lie group ${\rm{SO}}(p,q) = {\rm{SO}}(n)$ is connected. The crucial (non-trivial) fact is that the theory of connected compact Lie groups is completely "algebraic'', and in particular if $G$ and $H$ are two connected semisimple $\mathbf{R}$-groups for which $G(\mathbf{R})$ and $H(\mathbf{R})$ are compact then every Lie group homomorphism $G(\mathbf{R}) \rightarrow H(\mathbf{R})$ arising from a (unique) algebraic homomorphism $G \rightarrow H$. In particular, the automorphism groups of $G$ and $G(\mathbf{R})$ coincide, so the automorphism group of ${\rm{SO}}(n)$ coincides with that of $\mathbf{SO}(n)$.

Note that any linear automorphism preserving a non-degenerate quadratic form up to a nonzero scaling factor preserves its orthogonal and special orthogonal group. It is a general fact (due to Dieudonne over general fields away from characteristic 2) that if $(V, Q)$ is a non-degenerate quadratic space of dimension $n \ge 3$ over any field $k$ and if ${\mathbf{GO}}(Q)$ denotes the linear algebraic $k$-group of conformal automorphisms then the action of the algebraic group ${\mathbf{PGO}}(Q) = {\mathbf{GO}}(Q)/{\rm{GL}}_1$ on ${\mathbf{SO}}(Q)$ through conjugation gives exactly the automorphisms as an algebraic group. More specifically, $${\mathbf{PGO}}(Q)(k) = {\rm{Aut}}_k({\mathbf{SO}}(Q)).$$ This is proved using a lot of the structure theory of connected semisimple groups over an extension field that splits the quadratic form, so it is hard to "see'' this fact working directly over the given ground field $k$ (such as $k = \mathbf{R}$); that is one of the great merits of the algebraic theory (allowing us to prove results over a field by making calculations with a geometric object over an extension field, and using techniques such as Galois theory to come back to where we began).

---

Inside the automorphism group of the Lie group ${\rm{SO}}(p,q)$, we have built the subgroup ${\rm{PGO}}(p,q) := {\mathbf{PGO}}(p,q)(\mathbf{R})$ of "algebraic'' automorphisms (and it gives all automorphisms when $p$ or $q$ vanish). This subgroup is $${\mathbf{GO}}(p,q)(\mathbf{R})/\mathbf{R}^{\times} = {\rm{GO}}(p,q)/\mathbf{R}^{\times}.$$ To analyze the group ${\rm{GO}}(p,q)$ of conformal automorphisms of the quadratic space, there are two possibilities: if $p \ne q$ (such as whenever $p$ or $q$ vanish) then any such automorphism must involve a positive conformal scaling factor due to the need to preserve the signature, and if $p=q$ (the "split'' case: orthogonal sum of $p$ hyperbolic planes) then signature-preservation imposes no condition and we see (upon choosing a decomposition as an orthogonal sum of $p$ hyperbolic planes) that there is an evident involution $\tau$ of the vector space whose effect is to negative the quadratic form. Thus, if $p \ne q$ then ${\rm{GO}}(p,q) = \mathbf{R}^{\times} \cdot {\rm{O}}(p,q)$ whereas ${\rm{GO}}(p,p) = \langle \tau \rangle \ltimes (\mathbf{R}^{\times} \cdot {\rm{O}}(p,p))$. Hence, ${\rm{PGO}}(p,q) = {\rm{O}}(p,q)/\langle -1 \rangle$ if $p \ne q$ and ${\rm{PGO}}(p,p) = \langle \tau \rangle \ltimes ({\rm{O}}(p,p)/\langle -1 \rangle)$ for an explicit involution $\tau$ as above.

We summarize the conclusions for outer automorphisms of the Lie group ${\rm{SO}}(p, q)$ arising from the algebraic theory. If $n$ is odd (so $p \ne q$) then ${\rm{O}}(p,q) = \langle -1 \rangle \times {\rm{SO}}(p,q)$ and so the algebraic automorphisms are inner (as is very well-known in the algebraic theory). Suppose $n$ is even, so $-1 \in {\rm{SO}}(p, q)$. If $p \ne q$ (with the same parity) then the group of algebraic automorphisms contributes a subgroup of order 2 to the outer automorphism group (arising from any reflection in a non-isotropic vector, for example). Finally, the contribution of algebraic automorphisms to the outer automorphism group of ${\rm{SO}}(p,p)$ has order 4 (generated by two elements of order 2: an involution $\tau$ as above and a reflection in a non-isotropic vector). This settles the definite case as promised (i.e., all automorphisms inner for odd $n$ and outer automorphism group of order 2 via a reflection for even $n$) since in such cases we know that all automorphisms are algebraic.

---

Now we may and do assume $p, q > 0$. Does ${\rm{SO}}(p, q)$ have any non-algebraic automorphisms? We will show that if $n \ge 3$ is odd (i.e., $p$ and $q$ have opposite parity) or if $p$ and $q$ are both odd then there are no non-algebraic automorphisms (so we would be done).

First, let's compute $\pi_0({\rm{SO}}(p,q))$ for any $n \ge 3$. By the spectral theorem, the maximal compact subgroups of ${\rm{O}}(p,q)$ are the conjugates of the evident subgroup ${\rm{O}}(q) \times {\rm{O}}(q)$ with 4 connected components, and one deduces in a similar way that the maximal compact subgroups of ${\rm{SO}}(p, q)$ are the conjugates of the evident subgroup $$\{(g,g') \in {\rm{O}}(p) \times {\rm{O}}(q)\,|\, \det(g) = \det(g')\}$$ with 2 connected components. For any Lie group $\mathscr{H}$ with finite component group (such as the group $G(\mathbf{R})$ for any linear algebraic group $G$ over $\mathbf{R}$), the maximal compact subgroups $K$ constitute a single conjugacy class (with every compact subgroup contained in one) and as a smooth manifold $\mathscr{H}$ is a direct product of such a subgroup against a Euclidean space (see Chapter XV, Theorem 3.1 of Hochschild's book "Structure of Lie groups'' for a proof). In particular, $\pi_0(\mathscr{H}) = \pi_0(K)$, so ${\rm{SO}}(p, q)$ has exactly 2 connected components for any $p, q > 0$.

Now assume $n$ is odd, and swap $p$ and $q$ if necessary (as we may) so that $p$ is odd and $q>0$ is even. For any $g \in {\rm{O}}(q) - {\rm{SO}}(q)$, the element $(-1, g) \in {\rm{SO}}(p, q)$ lies in the unique non-identity component. Since $n \ge 3$ is odd, so ${\rm{SO}}(p, q)^0$ is the quotient of the connected (!) Lie group ${\rm{Spin}}(p, q)$ modulo its order-2 center, the algebraic theory in characteristic 0 gives $${\rm{Aut}}({\mathfrak{so}}(p,q)) = {\rm{Aut}}({\rm{Spin}}(p, q)) = {\rm{SO}}(p, q).$$ Thus, to find nontrivial elements of the outer automorphism group of the disconnected Lie group ${\rm{SO}}(p, q)$ we can focus attention on automorphisms $f$ of ${\rm{SO}}(p, q)$ that induce the identity on ${\rm{SO}}(p, q)^0$.

We have arranged that $p$ is odd and $q>0$ is even (so $q \ge 2$). The elements $$(-1, g) \in {\rm{SO}}(p, q) \cap ({\rm{O}}(p) \times {\rm{O}}(q))$$ (intersection inside ${\rm{O}}(p, q)$, so $g \in {\rm{O}}(q) - {\rm{SO}}(q)$) have an intrinsic characterization in terms of the Lie group ${\rm{SO}}(p, q)$ and its evident subgroups ${\rm{SO}}(p)$ and ${\rm{SO}}(q)$: these are the elements outside ${\rm{SO}}(p, q)^0$ that centralize ${\rm{SO}}(p)$ and normalize ${\rm{SO}}(q)$. (To prove this, consider the standard representation of ${\rm{SO}}(p) \times {\rm{SO}}(q)$ on $\mathbf{R}^{p+q} = \mathbf{R}^n$, especially the isotypic subspaces for the action of ${\rm{SO}}(q)$ with $q \ge 2$.) Hence, for every $g \in {\rm{O}}(q) - {\rm{SO}}(q)$ we have $f(-1,g) = (-1, F(g))$ for a diffeomorphism $F$ of the connected manifold ${\rm{O}}(q) - {\rm{SO}}(q)$.

Since $f$ acts as the identity on ${\rm{SO}}(q)$, it follows that the elements $g, F(g) \in {\rm{O}}(q) - {\rm{SO}}(q)$ have the same conjugation action on ${\rm{SO}}(q)$. But ${\rm{PGO}}(q) \subset {\rm{Aut}}({\rm{SO}}(q))$, so $F(g)g^{-1} \in \mathbf{R}^{\times}$ inside ${\rm{GL}}_q(\mathbf{R})$ with $q>0$ even. Taking determinants, this forces $F(g) = \pm g$ for a sign that may depend on $g$. But $F$ is continuous on the connected space ${\rm{O}}(q) - {\rm{SO}}(q)$, so the sign is actually independent of $g$. The case $F(g) = g$ corresponds to the identity automorphism of ${\rm{SO}}(q)$, so for the study of non-algebraic contributions to the outer automorphism group of ${\rm{SO}}(p, q)$ (with $p$ odd and $q > 0$ even) we are reduced to showing that the case $F(g) = -g$ cannot occur.

We are seeking to rule out the existence of an automorphism $f$ of ${\rm{SO}}(p, q)$ that is the identity on ${\rm{SO}}(p, q)^0$ and satisfies $(-1, g) \mapsto (-1, -g)$ for $g \in {\rm{O}}(q) - {\rm{SO}}(q)$. For this to be a homomorphism, it is necessary (and sufficient) that the conjugation actions of $(-1, g)$ and $(-1, -g)$ on ${\rm{SO}}(p, q)^0$ coincide for all $g \in {\rm{O}}(q) - {\rm{SO}}(q)$. In other words, this requires that the element $(1, -1) \in {\rm{SO}}(p, q)$ centralizes ${\rm{SO}}(p, q)^0$. But the algebraic group ${\mathbf{SO}}(p, q)$ is connected (for the Zariski topology) with trivial center and the same Lie algebra as ${\rm{SO}}(p, q)^0$, so by consideration of the compatible algebraic and analytic adjoint representations we see that $(1, -1)$ cannot centralize ${\rm{SO}}(p, q)^0$. Thus, no non-algebraic automorphism of ${\rm{SO}}(p, q)$ exists in the indefinite case when $n \ge 3$ is odd.

Finally, suppose $p$ and $q$ are both odd, so ${\rm{SO}}(p,q)^0$ does not contain the element $-1 \in {\rm{SO}}(p,q)$ that generates the center of ${\rm{SO}}(p,q)$ (and even the center of ${\rm{O}}(p,q)$). Thus, we have ${\rm{SO}}(p,q) = {\rm{SO}}(p,q)^0 \times \langle -1 \rangle$ with ${\rm{SO}}(p,q)^0$ having trivial center. Any (analytic) automorphism of ${\rm{SO}}(p,q)$ clearly acts trivially on the order-2 center $\langle -1 \rangle$ and must preserve the identity component too, so such an automorphism is determined by its effect on the identity component. It suffices to show that every analytic automorphism $f$ of ${\rm{SO}}(p,q)^0$ arises from an algebraic automorphism of ${\rm{SO}}(p,q)$, as then all automorphisms of ${\rm{SO}}(p,q)$ would be algebraic (so the determination of the outer analytic automorphism group for $p, q$ odd follows as for the definite case with even $n \ge 4$).

By the theory of connected semisimple algebraic groups in characteristic 0, for any $p, q \ge 0$ with $p+q \ge 3$ every analytic automorphism of the connected (!) group ${\rm{Spin}}(p,q)$ is algebraic. Thus, it suffices to show that any automorphism $f$ of ${\rm{SO}}(p,q)^0$ lifts to an automorphism of the degree-2 cover $\pi:{\rm{Spin}}(p,q) \rightarrow {\rm{SO}}(p,q)^0$. (Beware that this degree-2 cover is not the universal cover if $p, q \ge 2$, as ${\rm{SO}}(p,q)^0$ has maximal compact subgroup ${\rm{SO}}(p) \times {\rm{SO}}(q)$ with fundamental group of order 4.) The Lie algebra automorphism ${\rm{Lie}}(f)$ of ${\mathfrak{so}}(p,q) = {\mathfrak{spin}}(p,q)$ arises from a unique algebraic automorphism of the group ${\mathbf{Spin}}(p,q)$ since this latter group is simply connected in the sense of algebraic groups. The induced automorphism of the group ${\rm{Spin}}(p,q)$ of $\mathbf{R}$-points does the job, since its compatibility with $f$ via $\pi$ can be checked on Lie algebras (as we are working with connected Lie groups).

This final argument also shows that the remaining problem for even $p, q \ge 2$ is to determine if any automorphism of ${\rm{SO}}(p,q)$ that is the identity map on ${\rm{SO}}(p,q)^0$ is itself the identity map. (If affirmative for such $p, q$ then the outer automorphism group of ${\rm{SO}}(p,q)$ is of order 2, and if negative then the outer automorphism group is bigger.)

Answer 2

Edit. nfcd has given an almost complete answer. Let me add a few missing cases below. Unfortunately, I don't get away using only elementary methods.

Final edit. I will regard the groups $G=O(p,q)$ or $G=SO(p,q)$ as Lie groups (in the $C^\infty$ setting), to avoid complications with e.g. $SO(2)$ (which would have uncountably many outer automorphisms as an abstract group). On the other hand, algebraic geometry tools do not suffice as we see in nfcd's answer.

There are basically two sources for outer automorphisms in this case.

1. The normaliser of $G$ in $GL(n, \mathbb R)$ could be larger than $G$ itself. In this case, one has to check which elements of $N_{GL(n)}(G)/G$ act by conjugation in a different way than elements of $G$ itself.

2. There exists a nontrivial homomorphism $\varphi$ from $G/G^0$ to the center $C(G)$. In this case, one has to check that $g\mapsto \varphi(g)\cdot g$ is bijective. In contrast to the first type, these automorphisms change the spectral decomposition of a matrix in $G\subset GL(n,\mathbb R)$ in most cases.

I could not prove in general that matrix groups have no other outer automorphisms. But a case by case proof will reveal that here, all are generated by the two types above.

---

Let's start with $O(n)$. It is generated by reflections on hyperplanes, that is, by elements $g$ with eigenvalues $\pm 1$, such that the $-1$-eigenspace is one-dimensional. They satisfy three properties: $g^2=e$, $g\ne e$, and $C(g)\cong O(1)\times O(n-1)$. A maximal subset of commuting reflections corresponds to a selection of orthogonal lines in $\mathbb R^n$. With some extra work one sees that all automorphisms mapping reflections to reflections are inner automorphisms.

The only other elements with similar properties are reflections on lines, which have a one-dimensional $1$-eigenspace. If those happen to be in $SO(n)$, they cannot generate $O(n)$, so $O(n)$ has no outer automorphisms. This happens when $n$ is odd. If $n$ is even, you have an automorphism $g\mapsto\det(g)\cdot g$ flipping both types of generators. For $n=2$, there is no difference between hyperplanes and lines, and so by the argument above, all automorphisms are inner. For $n\ge 4$ even, an inner automorphism does not change the multiplicities of eigenvalues, hence the automorphism above is outer. So $\mathrm{Out}(O(n))=\mathbb Z/2$ if $n$ is even and $n\ge 4$.

For $SO(n)$, you already noticed that elements $g\in O(n)\setminus SO(n)$ give automorphisms. Because $-g$ and $g$ induce the same automorphism, this does not give an outer automorphism if $n$ is odd. Indeed, $SO(n)$ is generated by reflections along hyperplanes for $n$ odd, so by an argument as above there are no outer automorphisms.

If $n$ is even, one can check that the automorphism is indeed an outer one by regarding a matrix composed of $\frac n2$ rotation blocks of small nonzero angle. Such a matrix specifies an orientation on $\mathbb R^n$ that is preserved by all inner automorphisms, but not by an element of $O(n)\setminus SO(n)$.

To see that there are no more outer automorphism of $SO(n)$, one notices that every automorphism lifts to the universal cover $\mathrm{Spin}(n)$. This group is semisimple, compact, connected and simply connected for $n\ge 3$, so its automorphism group is the symmetry group of the Dynkin diagram, which is $\mathbb Z/2$ except if $n=8$. For $n=8$, the automorphism group is the symmetric group on $3$ elements. One can check that of these, only two descend to $SO(8)$.

---

For $G=O(p,q)$ or $G=SO(p,q)$ with both $p\ne 0$, $q\ne 0$, we have the following key observation. For each outer automorphism of $G$ there exists an outer automorphism of $G$ that acts on a fixed maximal compact subgroup $K$. For let $\Phi\colon G\to G$ represent an outer automorphism. Then $\Phi(K)\subset G$ is a maximal subgroup of $K$, hence conjugate to $K$ by an inner automorphism of $G$. The composition represents the same outer automorphism and acts on $K$.

Now assume that $\Phi$ acts as an inner automorphism on $K$. By composing with composition by an appropriate element of $K$, we may assume that $\Phi$ acts as identity on $K$. We want to find all automorphisms $\Phi$ that act as identity on all of $K$. Note that the choice of $K$ corresponds to a choice of splitting $\mathbb R^{p,q}\cong\mathbb R^p\oplus\mathbb R^q$. The group $G$ is generated by $K$ and by one-parameter groups of hyperbolic rotations that act on the span of two unit vectors $v\in\mathbb R^p$ and $w\in\mathbb R^q$ as $\bigl(\begin{smallmatrix}\cosh t&\sinh t\\\sinh t&\cosh t\end{smallmatrix}\bigr)$. All these subgroups are conjugate to each other by elements of $K$. Each subgroup commutes with a subgroup of $K$ that is isomorphic to $K\cap(O(p-1)\times O(q-1))$, and that determines the plane spanned by $v$ and $w$. The speed of such a rotation can be measured using the Killing form, which in intrinsic, so $\Phi$ can not change its absolute value. The upshot is that the only nontrivial automorphism that acts trivially on $K$ but not on $G$ is conjugation by $(\pm 1,\mp 1)\in O(p)\times O(q)$. It is an inner automorphism of $G$ except if we are dealing with $SO(p,q)$ and both $p$ and $q$ are odd. In that last case, it is an odd product of conjugations by reflections, and we will encounter it again below.

So from now on we consider outer automorphisms of $K$ and see if we can extend them to $G$.

We start with $O(p,q)$, which is a bit easier. Its maximal compact subgroup is $K=O(p)\times O(q)$. As above, we choose as generators a set $\mathbb RP^{p-1}\sqcup\mathbb RP^{q-1}$ consisting of all reflections along hyperplanes in both groups. Each element commutes with a subgroup isomorphic to $O(p-1,q)$ or $O(p,q-1)$, respectively. Reflections (of all of $\mathbb R^{p,q}$) along lines in $\mathbb R^p$ or $\mathbb R^q$ have similar properties. This way, we get three nontrivial endomorphisms given by multiplying each group element by a locally constant homomorphism $O(p,q)\to\{1,-1\}$, which we will denote by $\det_p$, $\det_q$ and $\det=\det_p\cdot\det_q$, which restrict on $K$ to $\det_{O(p)}$, $\det_{O(q)}$ and $\det_{O(p)}\cdot\det_{O(q)}$. If $p$ is even, multiplication with $\det_p$ is bijective, and hence an outer automorphism. If $q$ is even, multiplication with $\det_q$ is an automorphism, and if $p+q$ is even, multiplication with $\det$ is an automorphism. Only if $p=q=1$, multiplication with $\det$ does not change the spectral decomposition of at least one element of $K$, and similar as in the case of $O(2)$ above, it corresponds to the outer automorphism that comes from swapping both copies of $\mathbb R^1$. Because there are no other sets of generators with similar properties, we have found all outer automorphisms of $O(p,q)$.

The maximal compact subgroup of $SO(p,q)$ is $K=S(O(p)\times O(q))=SO(p,q)\cap O(p+q)$. It has two connected components. The connected component of the identity is $K^0=SO(p)\times SO(q)$. As in the case of $SO(n)$, the only possible outer automorphisms are generated by conjugating with reflections $r$ in $O(p)$ or $O(q)$. If $p+q$ is odd, $-r\in S(O(p)\times O(q))$ has the same effect, so one gets an inner automorphism. If $p+q$ is even, one gets a nontrivial outer automorphism by an orientation argument as above for $SO(n)$. An odd number of these compose to conjugation by $(-1,1)\in O(p)\times O(q)$ considered above. And of course for $p=q$, you get additional ones swapping both factors as above. Note that none of these automorphisms changes the eigenvalues of the matrices.

It remains to check if there are outer automorhisms that only effect $$R=K\setminus K^0=S(O(p)\times O(q))\setminus(SO(p)\times SO(q))=(O(p)\setminus SO(p))\times(O(q)\setminus SO(q))\;.$$ Such an automorphism becomes inner when restricted to $K^0$, so by composing with an inner automorphism, we find a representative $\Phi$ that acts as identity on $K^0$. We note that $R$ contains products $r_p\circ r_q$ of a reflection in $O(p)$ with a reflection in $O(q)$. The only other element of $R$ that act in the same way by conjugation on $SO(p)\times SO(q)$ would be $-r_p\circ r_q$, so all $\Phi$ can do is multiply elements of $R$ by $-1$. This gives a nontrivial endomorphism that is an outer automorphism if and only if $p$, $q$ are even (which changes the eigenvalues of some matrices, hence is not in our list above).

To summarize, if $p\ne q$, the outer automorphism group is of the form $(\mathbb Z/2)^k$. Generators are given below. $$\begin{matrix} \text{group}&\text{case}&\text{generators}\\ O(n)&\text{$n$ odd or $n=2$}&\text{---}\\ O(n)&\text{$n$ even, $n\ge 4$}&\mu_{\det}\\ SO(n)&\text{$n$ odd}&\text{---}\\ SO(n)&\text{$n$ even}&C_r\\ O(p,q)&\text{$p$, $q$ odd, $p+q\ge 4$}&\mu_{\det}\\ O(p,q)&\text{$p$ even, $q$ odd}&\mu_{\det_p}\\ O(p,q)&\text{$p$, $q$ even}&\mu_{\det_p},\mu_{\det_q}\\ SO(p,q)&\text{$p$, $q$ odd}&C_r\\ SO(p,q)&\text{$p$ even, $q$ odd}&\text{---}\\ SO(p,q)&\text{$p$, $q$ even}&C_r,\mu_{\det_p} \end{matrix}$$ where $\mu_{\dots}$ denotes multiplication with a homomorphism to the center, $C_{\dots}$ denotes conjugation with an element in the normaliser, and $r$ denotes a reflection. If $p=q$, then there is an additional generator induced by swapping the two copies of $\mathbb R^p$. The full outer automorphism group will then be of the form $(\mathbb Z/2)^k\rtimes(\mathbb Z/2)$, where $(\mathbb Z/2)^k$ is the group described in the table.

Answer 3

This is discussed in this paper by Brian Roberts. (2010), where he points out that the outer automorphism group of the orthogonal groups is trivial.

Answer 4

on page 386 (paragraph 66.7) you find the table Out(G)/Int(G) on page 387 you find D_{l,j} j>1 your Lie algebras so(p,q) when your p or q is even. on page 391 you find so(p,q) when both p,q are odd The general theorem is on page 382-386

Answer 5

also check

Onishchik: Lectures on Real Semisimple Lie Algebras and Their Representations.

Answer 6

Check the book Freudenthal Linear Lie groups, in this book you find a complete treatment of Aut(G) for G a real semisimple (connected) Lie group. best regards