Return to Answer

Final edit. I will regard the groups $G=O(p,q)$ or $G=SO(p,q)$ as Lie groups (in the $C^\infty$ setting), to avoid complications with e.g. $SO(2)$ (which would have uncountably many outer automorphisms as an abstract group). On the other hand, algebraic tools do not suffice as we see in nfcd's answer.

We start with $O(p,q)$, which is a bit easier. Its maximal compact subgroup is $K=O(p)\times O(q)$. As above, we choose as generators a set $\mathbb RP^{p-1}\sqcup\mathbb RP^{q-1}$ consisting of all reflections along hyperplanes in both groups. Each element commutes with a subgroup isomorphic to $O(p-1,q)$ or $O(p,q-1)$, respectively. Reflections (of all of $\mathbb R^{p,q}$) along lines in $\mathbb R^p$ or $\mathbb R^q$ have similar properties. This way, we get three nontrivial endomorphisms given by multiplying each group element by a locally constant homomorphism $O(p,q)\to\{1,-1\}$, which we will denote by $\det_p$, $\det_q$ and $\det=\det_p\cdot\det_q$, which restrict on $K$ to $\det_{O(p)}$, $\det_{O(q)}$ and $\det_{O(p)}\cdot\det_{O(q)}$. If $p$ is even, multiplication with $\det_p$ is bijective, and hence an outer automorphism. If $q$ is even, multiplication with $\det_q$ is an automorphism, and if $p+q$ is even, multiplication with $\det$ is an automorphism. Only if $p=q=1$, multiplication with $\det$ does not change the spectral decomposition of at least one element of $K$, and similar as in the case of $O(2)$ above, it corresponds to the outer automorphism that comes from swapping both copies of $\R^1$. Because there are no other sets of generators with similar properties, we have found all outer automorphisms of $O(p,q)$.

To summarize, if $p\ne q$, the outer automorphism group is of the form $(\mathbb Z/2)^k$. Generators are given below. $$\begin{matrix} \text{group}&\text{case}&\text{generators}\\ O(n)&\text{$n$ odd or $n=2$}&\text{---}\\ O(n)&\text{$n$ even, $n\ge 4$}&\mu_{\det}\\ SO(n)&\text{$n$ odd}&\text{---}\\ SO(n)&\text{$n$ even}&C_r\\ O(p,q)&\text{$p$, $q$ odd, $p+q\ge 4$}&\mu_{\det}\\ O(p,q)&\text{$p$ even, $q$ odd}&\mu_{\det_p}\\ O(p,q)&\text{$p$, $q$ even}&\mu_{\det_p},\mu_{\det_q}\\ SO(p,q)&\text{$p$, $q$ odd}&C_r\\ SO(p,q)&\text{$p$ even, $q$ odd}&\text{---}\\ SO(p,q)&\text{$p$, $q$ even}&C_r,\mu_{\det_p} \end{matrix}$$ where $\mu_{\dots}$ denotes multiplication with a homomorphism to the center, $C_{\dots}$ denotes conjugation with an element in the normaliser, and $r$ denotes a reflection. If $p=q$, then there is an additional generator induced by swapping the two copies of $\R^p$. The full outer automorphism group will then be of the form $(\mathbb Z/2)^k\rtimes(\mathbb Z/2)$, where $(\mathbb Z/2)^k$ is the group described in the table.

Final edit. I will regard the groups $G=O(p,q)$ or $G=SO(p,q)$ as Lie groups (in the $C^\infty$ setting), to avoid complications with e.g. $SO(2)$ (which would have uncountably many outer automorphisms as an abstract group). On the other hand, algebraic geometry tools do not suffice as we see in nfcd's answer.

We start with $O(p,q)$, which is a bit easier. Its maximal compact subgroup is $K=O(p)\times O(q)$. As above, we choose as generators a set $\mathbb RP^{p-1}\sqcup\mathbb RP^{q-1}$ consisting of all reflections along hyperplanes in both groups. Each element commutes with a subgroup isomorphic to $O(p-1,q)$ or $O(p,q-1)$, respectively. Reflections (of all of $\mathbb R^{p,q}$) along lines in $\mathbb R^p$ or $\mathbb R^q$ have similar properties. This way, we get three nontrivial endomorphisms given by multiplying each group element by a locally constant homomorphism $O(p,q)\to\{1,-1\}$, which we will denote by $\det_p$, $\det_q$ and $\det=\det_p\cdot\det_q$, which restrict on $K$ to $\det_{O(p)}$, $\det_{O(q)}$ and $\det_{O(p)}\cdot\det_{O(q)}$. If $p$ is even, multiplication with $\det_p$ is bijective, and hence an outer automorphism. If $q$ is even, multiplication with $\det_q$ is an automorphism, and if $p+q$ is even, multiplication with $\det$ is an automorphism. Only if $p=q=1$, multiplication with $\det$ does not change the spectral decomposition of at least one element of $K$, and similar as in the case of $O(2)$ above, it corresponds to the outer automorphism that comes from swapping both copies of $\mathbb R^1$. Because there are no other sets of generators with similar properties, we have found all outer automorphisms of $O(p,q)$.

To summarize, if $p\ne q$, the outer automorphism group is of the form $(\mathbb Z/2)^k$. Generators are given below. $$\begin{matrix} \text{group}&\text{case}&\text{generators}\\ O(n)&\text{$n$ odd or $n=2$}&\text{---}\\ O(n)&\text{$n$ even, $n\ge 4$}&\mu_{\det}\\ SO(n)&\text{$n$ odd}&\text{---}\\ SO(n)&\text{$n$ even}&C_r\\ O(p,q)&\text{$p$, $q$ odd, $p+q\ge 4$}&\mu_{\det}\\ O(p,q)&\text{$p$ even, $q$ odd}&\mu_{\det_p}\\ O(p,q)&\text{$p$, $q$ even}&\mu_{\det_p},\mu_{\det_q}\\ SO(p,q)&\text{$p$, $q$ odd}&C_r\\ SO(p,q)&\text{$p$ even, $q$ odd}&\text{---}\\ SO(p,q)&\text{$p$, $q$ even}&C_r,\mu_{\det_p} \end{matrix}$$ where $\mu_{\dots}$ denotes multiplication with a homomorphism to the center, $C_{\dots}$ denotes conjugation with an element in the normaliser, and $r$ denotes a reflection. If $p=q$, then there is an additional generator induced by swapping the two copies of $\mathbb R^p$. The full outer automorphism group will then be of the form $(\mathbb Z/2)^k\rtimes(\mathbb Z/2)$, where $(\mathbb Z/2)^k$ is the group described in the table.

Edit. nfcd has given an almost complete answer. Let me add a few missing cases below and clarify some of the statements from the first version. Unfortunately, I don't get away using only elementary methods.

(Hopefully) final edit. There were a few problems, that I've now hopefully corrected.

I will regard the groups $O(p,q)$ and $SO(p,q)$ as Lie groups (in the $C^\infty$ setting), to avoid complications with e.g. $SO(2)$ (which would have uncountably many outer automorphisms as an abstract group). There are basically two sources for outer automorphisms in this case.

1. The normaliser of $G$ in $GL(n, \mathbb R)$ could be larger than $G$ itself. In this case, one has to check which elements of $N_{GL(n)}(G)/G$ act by conjugation in a different way than elements of $G$ itself.

2. There exists a nontrivial homomorphism $\varphi$ from $G/G^0$ to the center $C(G)$. In this case, one has to check that $g\mapsto \varphi(g)\cdot g$ is bijective. In contrast to the first type, these automorphisms change the spectral decomposition of a matrix in $G\subset GL(n,\mathbb R)$.

I could not prove generally that there cannot be other outer automorphisms. But a case by case proof will reveal that all are generated by the two types above.

We start with $O(p,q)$, which is a bit easier. Its maximal compact subgroup is $K=O(p)\times O(q)$. As above, we choose as generators a set $\mathbb RP^{p-1}\sqcup\mathbb RP^{q-1}$ consisting of all reflections along hyperplanes in both groups. Each element commutes with a subgroup isomorphic to $O(p-1,q)$ or $O(p,q-1)$, respectively. Reflections (of all of $\mathbb R^{p,q}$) along lines in $\mathbb R^p$ or $\mathbb R^q$ have similar properties. This way, we get three nontrivial endomorphisms given by multiplying each group element by a locally constant homomorphism $O(p,q)\to\{1,-1\}$, which we will denote by $\det_{\mathbb R^p}$, $\det_{\mathbb R^q}$ and $\det=\det_{\mathbb R^p}\cdot\det_{\mathbb R^q}$. If $p$ is even, multiplication with $\det_{\mathbb R^p}$ is bijective, and hence an outer automorphism. If $q$ is even, multiplication with $\det_{\mathbb R^q}$ is an automorphism, and if $p+q$ is even, multiplication with $\det$ is an automorphism. Because there are no other sets of generators with similar properties, we have found all outer automorphisms of $O(p,q)$.

It remains to check if there are outer automorhisms that only effect $$R=K\setminus K^0=S(O(p)\times O(q))\setminus(SO(p)\times SO(q))=(O(p)\setminus SO(p))\times(O(q)\setminus SO(q))\;.$$ Such an automorphism becomes inner when restricted to $K^0$, so by composing with an inner automorphism, we find a representative $\Phi$ that acts as identity on $K^0$. We note that $R$ contains products $r_p\circ r_q$ of a reflection in $O(p)$ with a reflection in $O(q)$. The only other element of $R$ that act in the same way by conjugation on $SO(p)\times SO(q)$ would be $-r_p\circ r_q$, so all $\Phi$ can do is multiply elements of $R$ by $-1$. This gives a nontrivial endomorphism that is an outer automorphism if and only if $p+q$ is even (which changes the eigenvalues of some matrices, hence is not in our list above).

To summarize, if $p\ne q$, the outer automorphism group is of the form $(\mathbb Z/2)^k$. Generators restrict on $K$ as follows. $$\begin{matrix} \text{group}&\text{case}&\text{generators}\\ O(n)&\text{$n$ odd}&\text{---}\\ O(n)&\text{$n$ even}&\mu_{\det}\\ SO(n)&\text{$n$ odd}&\text{---}\\ SO(n)&\text{$n$ even}&C_r\\ O(p,q)&\text{$p$, $q$ odd}&\mu_{\det}\\ O(p,q)&\text{$p$ even, $q$ odd}&\mu_{\det_{\mathbb R^p}}\\ O(p,q)&\text{$p$, $q$ even}&\mu_{\det_{\mathbb R^p}},\mu_{\det_{\mathbb R^q}}\\ SO(p,q)&\text{$p$, $q$ odd}&C_r,\mu_{\det_{\mathbb R^p}}\\ SO(p,q)&\text{$p$ even, $q$ odd}&\text{---}\\ SO(p,q)&\text{$p$, $q$ even}&C_r,\mu_{\det_{\mathbb R^p}} \end{matrix}$$ where $\mu_{\dots}$ denotes multiplication with a homomorphism to the center, $C_{\dots}$ denotes conjugation with an element in the normaliser, and $r$ denotes a reflection.

Edit. nfcd has given an almost complete answer. Let me add a few missing cases below. Unfortunately, I don't get away using only elementary methods.

Final edit. I will regard the groups $G=O(p,q)$ or $G=SO(p,q)$ as Lie groups (in the $C^\infty$ setting), to avoid complications with e.g. $SO(2)$ (which would have uncountably many outer automorphisms as an abstract group). On the other hand, algebraic tools do not suffice as we see in nfcd's answer.

There are basically two sources for outer automorphisms in this case.

1. The normaliser of $G$ in $GL(n, \mathbb R)$ could be larger than $G$ itself. In this case, one has to check which elements of $N_{GL(n)}(G)/G$ act by conjugation in a different way than elements of $G$ itself.

2. There exists a nontrivial homomorphism $\varphi$ from $G/G^0$ to the center $C(G)$. In this case, one has to check that $g\mapsto \varphi(g)\cdot g$ is bijective. In contrast to the first type, these automorphisms change the spectral decomposition of a matrix in $G\subset GL(n,\mathbb R)$ in most cases.

I could not prove in general that matrix groups have no other outer automorphisms. But a case by case proof will reveal that here, all are generated by the two types above.

We start with $O(p,q)$, which is a bit easier. Its maximal compact subgroup is $K=O(p)\times O(q)$. As above, we choose as generators a set $\mathbb RP^{p-1}\sqcup\mathbb RP^{q-1}$ consisting of all reflections along hyperplanes in both groups. Each element commutes with a subgroup isomorphic to $O(p-1,q)$ or $O(p,q-1)$, respectively. Reflections (of all of $\mathbb R^{p,q}$) along lines in $\mathbb R^p$ or $\mathbb R^q$ have similar properties. This way, we get three nontrivial endomorphisms given by multiplying each group element by a locally constant homomorphism $O(p,q)\to\{1,-1\}$, which we will denote by $\det_p$, $\det_q$ and $\det=\det_p\cdot\det_q$, which restrict on $K$ to $\det_{O(p)}$, $\det_{O(q)}$ and $\det_{O(p)}\cdot\det_{O(q)}$. If $p$ is even, multiplication with $\det_p$ is bijective, and hence an outer automorphism. If $q$ is even, multiplication with $\det_q$ is an automorphism, and if $p+q$ is even, multiplication with $\det$ is an automorphism. Only if $p=q=1$, multiplication with $\det$ does not change the spectral decomposition of at least one element of $K$, and similar as in the case of $O(2)$ above, it corresponds to the outer automorphism that comes from swapping both copies of $\R^1$. Because there are no other sets of generators with similar properties, we have found all outer automorphisms of $O(p,q)$.

It remains to check if there are outer automorhisms that only effect $$R=K\setminus K^0=S(O(p)\times O(q))\setminus(SO(p)\times SO(q))=(O(p)\setminus SO(p))\times(O(q)\setminus SO(q))\;.$$ Such an automorphism becomes inner when restricted to $K^0$, so by composing with an inner automorphism, we find a representative $\Phi$ that acts as identity on $K^0$. We note that $R$ contains products $r_p\circ r_q$ of a reflection in $O(p)$ with a reflection in $O(q)$. The only other element of $R$ that act in the same way by conjugation on $SO(p)\times SO(q)$ would be $-r_p\circ r_q$, so all $\Phi$ can do is multiply elements of $R$ by $-1$. This gives a nontrivial endomorphism that is an outer automorphism if and only if $p$, $q$ are even (which changes the eigenvalues of some matrices, hence is not in our list above).

To summarize, if $p\ne q$, the outer automorphism group is of the form $(\mathbb Z/2)^k$. Generators are given below. $$\begin{matrix} \text{group}&\text{case}&\text{generators}\\ O(n)&\text{$n$ odd or $n=2$}&\text{---}\\ O(n)&\text{$n$ even, $n\ge 4$}&\mu_{\det}\\ SO(n)&\text{$n$ odd}&\text{---}\\ SO(n)&\text{$n$ even}&C_r\\ O(p,q)&\text{$p$, $q$ odd, $p+q\ge 4$}&\mu_{\det}\\ O(p,q)&\text{$p$ even, $q$ odd}&\mu_{\det_p}\\ O(p,q)&\text{$p$, $q$ even}&\mu_{\det_p},\mu_{\det_q}\\ SO(p,q)&\text{$p$, $q$ odd}&C_r\\ SO(p,q)&\text{$p$ even, $q$ odd}&\text{---}\\ SO(p,q)&\text{$p$, $q$ even}&C_r,\mu_{\det_p} \end{matrix}$$ where $\mu_{\dots}$ denotes multiplication with a homomorphism to the center, $C_{\dots}$ denotes conjugation with an element in the normaliser, and $r$ denotes a reflection. If $p=q$, then there is an additional generator induced by swapping the two copies of $\R^p$. The full outer automorphism group will then be of the form $(\mathbb Z/2)^k\rtimes(\mathbb Z/2)$, where $(\mathbb Z/2)^k$ is the group described in the table.

Let's start with $O(n)$. It is generated by reflections on hyperplanes, that is, by elements $g$ with eigenvalues $\pm 1$, such that the $-1$-eigenspace is one-dimensional. They satisfy three properties: $g^2=e$, $g\ne e$, and $C(g)\cong O(1)\times O(n-1)$. A maximal subset of commuting reflections corresponds to a selection of orthogonal lines in $\mathbb R^n$. With some extra work one sees that all automorphisms mapping reflections to reflections are inner automorphisms.

Now assume that $\Phi$ acts as an inner automorphism on $K$. By composing with composition by an appropriate element of $K$, we may assume that $\Phi$ acts as identity on $K$. We want to prove that $\Phi$ acts as identity on all of $G$. Note that the choice of $K$ corresponds to a choice of splitting $\mathbb R^{p,q}\cong\mathbb R^p\oplus\mathbb R^q$. The group $G$ is generated by $K$ and by one-parameter groups of hyperbolic rotations that act on the span of two unit vectors $v\in\mathbb R^p$ and $w\in\mathbb R^q$ as $\bigl(\begin{smallmatrix}\cosh t&\sinh t\\\sinh t&\cosh t\end{smallmatrix}\bigr)$. All these subgroups are conjugate to each other by elements of $K$. Each subgroup commutes with a subgroup of $K$ that is isomorphic to $K\cap(O(p-1)\times O(q-1))$, and which determines $v$ and $w$ up to sign. The speed of such a rotation can be measured using the Killing form, which in intrinsic, so $\Phi$ can not change its absolute value. The upshot is that the only nontrivial automorphism that acts trivially on $K$ but not on $G$ is conjugation by $(\pm 1,\mp 1)\in O(p)\times O(q)$. It is an inner automorphism of $G$ except if we are dealing with $SO(p,q)$ and both $p$ and $q$ are odd.

We start with $O(p,q)$, which is a bit easier. Its maximal compact subgroup is $K=O(p)\times O(q)$. We can construct nontrivial automorphisms e.g. by $(g,h)\mapsto\det g\cdot (g,h)$. This easily extends to $O(p,q)$ because $\det g$, $\det h$ extend to locally constant functions on $O(p,q)$. It remains to see that these are all. If $p=q$ is odd, swapping the factors generates the outer automorphism group $\mathbb Z/2$. If $p=q$ is even, we get $(\mathbb Z/2)^2\rtimes\mathbb Z/2$.

The maximal compact subgroup of $SO(p,q)$ is $K=S(O(p)\times O(q))=SO(p,q)\cap O(p+q)$. It has two connected components. The connected component of the identity is $K^0=SO(p)\times SO(q)$, and we know its outer automorphisms from above. Because they are induced by inner automorphisms of $O(p)$ or $O(q)$ respectively, they extend to $SO(p,q)$.

However, they could become inner automorphisms. Indeed, as pointed out in nfcd's answer, if $p$ is odd and $q$ is even, conjugation with $(1,h)$ for $h\in O(q)\setminus SO(q)$ has the same effect as conjugation with $(-1,h)\in SO(p,q)$. Moreover, if both $p$ and $q$ are even, then two of the four automotphisms become inner: $(g,h)\in SO(p,q)$ if~$g\in O(p)\setminus SO(p)$ and $h\in O(q)\setminus SO(q)$. The remaining nontrivial one is an outer automorphism by an orientation argument as above for $SO(n)$. And of course for $p=q$, you get additional ones as above. Note that none of these automorphisms changes the eigenvalues of the matrices.

It remains to check if there are outer automorhisms that only effect $$R=K\setminus K^0=S(O(p)\times O(q))\setminus(SO(p)\times SO(q))=(O(p)\setminus SO(p))\times(O(q)\setminus SO(q))\;.$$ Such an automorphism becomes inner when restricted to $K^0$, so by composing with an inner automorphism, we find a representative $\Phi$ that acts as identity on $K^0$. If both $p$ and $q$ are even, then $-1\in SO(p)\times SO(q)$ is central of order two, so multiplying each element of $R$ with $-1$ induces an outer isomorphism (which changes the eigenvalues of some matrices, hence is not in our list above). In all other cases, we note that $R$ contains products of a reflection in $O(p)$ with a reflection in $O(q)$. Then all that $\Phi$ can do is replace one or both by the reflection on the perpendicular hyperplane. But if only one of the two factors is effected, there is no way of extending $\Phi$ to all of $SO(p,q)$, because we would have to split $\mathbb R^{p+q}=\mathbb R^p\oplus\mathbb R^q$ artificially.

To summarize, if $p\ne q$, the outer automorphism group is of the form $(\mathbb Z/2)^k$. Generators restrict on $K$ as follows. $$\begin{matrix} \text{group}&\text{case}&\text{generators}\\ O(p,q)&\text{$p$, $q$ odd}&\text{---}\\ O(p,q)&\text{$p$ even, $q$ odd}&(g,h)\mapsto\det g\cdot (g,h)\\ O(p,q)&\text{$p$, $q$ even}&(g,h)\mapsto\det g\cdot (g,h),\quad(g,h)\mapsto\det h\cdot (g,h)\\ SO(p,q)&\text{$p$, $q$ odd}&C_{(1,-1)}\\ SO(p,q)&\text{$p$ even, $q$ odd}&\text{---}\\ SO(p,q)&\text{$p$, $q$ even}&C_r,\quad(g,h)\mapsto\det g\cdot(g,h) \end{matrix}$$ where $C_\dots$ denotes conjugation, and $r$ denotes a reflection.

(Hopefully) final edit. There were a few problems, that I've now hopefully corrected.

I will regard the groups $O(p,q)$ and $SO(p,q)$ as Lie groups (in the $C^\infty$ setting), to avoid complications with e.g. $SO(2)$ (which would have uncountably many outer automorphisms as an abstract group). There are basically two sources for outer automorphisms in this case.

1. The normaliser of $G$ in $GL(n, \mathbb R)$ could be larger than $G$ itself. In this case, one has to check which elements of $N_{GL(n)}(G)/G$ act by conjugation in a different way than elements of $G$ itself.

2. There exists a nontrivial homomorphism $\varphi$ from $G/G^0$ to the center $C(G)$. In this case, one has to check that $g\mapsto \varphi(g)\cdot g$ is bijective. In contrast to the first type, these automorphisms change the spectral decomposition of a matrix in $G\subset GL(n,\mathbb R)$.

I could not prove generally that there cannot be other outer automorphisms. But a case by case proof will reveal that all are generated by the two types above.

Let's start with $O(n)$. It is generated by reflections on hyperplanes, that is, by elements $g$ with eigenvalues $\pm 1$, such that the $-1$-eigenspace is one-dimensional. They satisfy three properties: $g^2=e$, $g\ne e$, and $C(g)\cong O(1)\times O(n-1)$. A maximal subset of commuting reflections corresponds to a selection of orthogonal lines in $\mathbb R^n$. With some extra work one sees that all automorphisms mapping reflections to reflections are inner automorphisms.

Now assume that $\Phi$ acts as an inner automorphism on $K$. By composing with composition by an appropriate element of $K$, we may assume that $\Phi$ acts as identity on $K$. We want to find all automorphisms $\Phi$ that act as identity on all of $K$. Note that the choice of $K$ corresponds to a choice of splitting $\mathbb R^{p,q}\cong\mathbb R^p\oplus\mathbb R^q$. The group $G$ is generated by $K$ and by one-parameter groups of hyperbolic rotations that act on the span of two unit vectors $v\in\mathbb R^p$ and $w\in\mathbb R^q$ as $\bigl(\begin{smallmatrix}\cosh t&\sinh t\\\sinh t&\cosh t\end{smallmatrix}\bigr)$. All these subgroups are conjugate to each other by elements of $K$. Each subgroup commutes with a subgroup of $K$ that is isomorphic to $K\cap(O(p-1)\times O(q-1))$, and that determines the plane spanned by $v$ and $w$. The speed of such a rotation can be measured using the Killing form, which in intrinsic, so $\Phi$ can not change its absolute value. The upshot is that the only nontrivial automorphism that acts trivially on $K$ but not on $G$ is conjugation by $(\pm 1,\mp 1)\in O(p)\times O(q)$. It is an inner automorphism of $G$ except if we are dealing with $SO(p,q)$ and both $p$ and $q$ are odd. In that last case, it is an odd product of conjugations by reflections, and we will encounter it again below.

We start with $O(p,q)$, which is a bit easier. Its maximal compact subgroup is $K=O(p)\times O(q)$. As above, we choose as generators a set $\mathbb RP^{p-1}\sqcup\mathbb RP^{q-1}$ consisting of all reflections along hyperplanes in both groups. Each element commutes with a subgroup isomorphic to $O(p-1,q)$ or $O(p,q-1)$, respectively. Reflections (of all of $\mathbb R^{p,q}$) along lines in $\mathbb R^p$ or $\mathbb R^q$ have similar properties. This way, we get three nontrivial endomorphisms given by multiplying each group element by a locally constant homomorphism $O(p,q)\to\{1,-1\}$, which we will denote by $\det_{\mathbb R^p}$, $\det_{\mathbb R^q}$ and $\det=\det_{\mathbb R^p}\cdot\det_{\mathbb R^q}$. If $p$ is even, multiplication with $\det_{\mathbb R^p}$ is bijective, and hence an outer automorphism. If $q$ is even, multiplication with $\det_{\mathbb R^q}$ is an automorphism, and if $p+q$ is even, multiplication with $\det$ is an automorphism. Because there are no other sets of generators with similar properties, we have found all outer automorphisms of $O(p,q)$.

The maximal compact subgroup of $SO(p,q)$ is $K=S(O(p)\times O(q))=SO(p,q)\cap O(p+q)$. It has two connected components. The connected component of the identity is $K^0=SO(p)\times SO(q)$. As in the case of $SO(n)$, the only possible outer automorphisms are generated by conjugating with reflections $r$ in $O(p)$ or $O(q)$. If $p+q$ is odd, $-r\in S(O(p)\times O(q))$ has the same effect, so one gets an inner automorphism. If $p+q$ is even, one gets a nontrivial outer automorphism by an orientation argument as above for $SO(n)$. An odd number of these compose to conjugation by $(-1,1)\in O(p)\times O(q)$ considered above. And of course for $p=q$, you get additional ones swapping both factors as above. Note that none of these automorphisms changes the eigenvalues of the matrices.

It remains to check if there are outer automorhisms that only effect $$R=K\setminus K^0=S(O(p)\times O(q))\setminus(SO(p)\times SO(q))=(O(p)\setminus SO(p))\times(O(q)\setminus SO(q))\;.$$ Such an automorphism becomes inner when restricted to $K^0$, so by composing with an inner automorphism, we find a representative $\Phi$ that acts as identity on $K^0$. We note that $R$ contains products $r_p\circ r_q$ of a reflection in $O(p)$ with a reflection in $O(q)$. The only other element of $R$ that act in the same way by conjugation on $SO(p)\times SO(q)$ would be $-r_p\circ r_q$, so all $\Phi$ can do is multiply elements of $R$ by $-1$. This gives a nontrivial endomorphism that is an outer automorphism if and only if $p+q$ is even (which changes the eigenvalues of some matrices, hence is not in our list above).

To summarize, if $p\ne q$, the outer automorphism group is of the form $(\mathbb Z/2)^k$. Generators restrict on $K$ as follows. $$\begin{matrix} \text{group}&\text{case}&\text{generators}\\ O(n)&\text{$n$ odd}&\text{---}\\ O(n)&\text{$n$ even}&\mu_{\det}\\ SO(n)&\text{$n$ odd}&\text{---}\\ SO(n)&\text{$n$ even}&C_r\\ O(p,q)&\text{$p$, $q$ odd}&\mu_{\det}\\ O(p,q)&\text{$p$ even, $q$ odd}&\mu_{\det_{\mathbb R^p}}\\ O(p,q)&\text{$p$, $q$ even}&\mu_{\det_{\mathbb R^p}},\mu_{\det_{\mathbb R^q}}\\ SO(p,q)&\text{$p$, $q$ odd}&C_r,\mu_{\det_{\mathbb R^p}}\\ SO(p,q)&\text{$p$ even, $q$ odd}&\text{---}\\ SO(p,q)&\text{$p$, $q$ even}&C_r,\mu_{\det_{\mathbb R^p}} \end{matrix}$$ where $\mu_{\dots}$ denotes multiplication with a homomorphism to the center, $C_{\dots}$ denotes conjugation with an element in the normaliser, and $r$ denotes a reflection.