How are orthogonal groups over $\mathbb{R}$ parameterized by symmetric bilinear forms?

Question

Let $G$ be an algebraic group over $\mathbb{C}$. Suppose $G$ is given an $\mathbb{R}$-structure, i.e. an $\mathbb{R}$-algebra $A_0$ such that $A = \mathbb{C} \otimes_{\mathbb{R}} A_0$, where $A$ is the coordinate ring of $G$. Then the Galois group $\Gamma = \textrm{Gal}(\mathbb{C}/\mathbb{R})$ acts as a group of ring automorphisms $A$ as $\sigma.(\lambda \otimes a) = \sigma(\lambda) \otimes a$, which induces an action of $\Gamma$ on $\textrm{Spm } A = G$.

Then $\Gamma$ further acts on $\textrm{Aut}(G)$, the group of algebraic group automorphisms of $G$, as $\gamma.\phi(x) = \gamma^{-1}.\phi(\gamma.x)$

The isomorphism classes of $\mathbb{R}$-structures on $G$ form a pointed set, the distinguished element corresponding to class of $A_0$. This pointed set is isomorphic to $H^1(\Gamma, \textrm{Aut}(G))$, the set of equivalence classes of one-cocycles, i.e. functions $c: \Gamma \rightarrow \textrm{Aut}(G)$ for which $c(\gamma_1\gamma_2) = c(\gamma_1) \circ \gamma_1.(c(\gamma_2))$ for all $\gamma_i \in \Gamma$.

If $M$ is a symmetric invertible real matrix, one can define $\textrm{O}_n(\mathbb{C}) = \{ x \in \textrm{GL}_n(\mathbb{C}) : x^t Mx = M \}$. As an algebraic group over $\mathbb{C}$, $O_n(\mathbb{C})$ does not depend on the choice of $M$ up to isomorphism. But over $\mathbb{R}$, different choice of $M$ may yield different groups.

Is there is a connection between the set $H^1(\Gamma, \textrm{Aut}(G))$ and classes of invertible symmetric matrices $M$? (two matrices $M_1, M_2$ being equivalent if there exists an $x \in \textrm{GL}_n(\mathbb{R})$ such that $x^tM_1x = M_2$)

Answer 1

For $\mathit{SO}_n$ instead of $\mathit{O}_n$, and for real semisimple groups in general, your question is quite clearly addressed by the text "Galois Cohomology of Real Groups" by Jeffrey Adams (see, in particular, the tables given at the end). See also these slides by Mikhail Borovoi. This should at least clarify a number of things about Galois cohomology of real algebraic groups.

The classes of rank $n$ real quadratic forms (i.e., equivalence classes of symmetric matrices) are classified by $H^1(\mathbb{R},\mathit{O}_n)$, which has cardinality $n+1$ given by Sylvester's law of inertia (the signature of the quadratic form). The real forms of $\mathit{O}_n$ are, as you remark, classified by $H^1(\mathbb{R}, \mathrm{Aut}(\mathit{O}_n))$. Now we have an inner automorphism map $\mathit{O}_n \to \mathrm{Aut}(\mathit{O}_n)$ whose kernel has order $2$ and whose image has index $1$ or $2$ (the latter if $n$ is even $\geq 4$, I hope I got this right: see here). From Galois cohomology we get a map $H^1(\mathbb{R},\mathit{O}_n) \to H^1(\mathbb{R}, \mathrm{Aut}(\mathit{O}_n))$ (of pointed sets) which, concretely, maps a quadratic form to the orthogonal group it defines, which factors through $H^1(\mathbb{R}, \mathrm{Inn}(\mathit{O}_n))$ where $\mathrm{Inn}(\mathit{O}_n) = \mathit{O}_n/\{\pm1\}$. So certainly there "is a connection". But it is not quite a bijection.

For $n$ odd, $\mathrm{Aut}(\mathit{O}_n) = \mathrm{Inn}(\mathit{O}_n) = \mathit{SO}_n$, the set $H^1(\mathbb{R}, \mathit{SO}_n)$ has cardinality $\frac{n+1}{2}$ (see Adams's paper mentioned above), and the map $H^1(\mathbb{R},\mathit{O}_n) \to H^1(\mathbb{R}, \mathrm{Aut}(\mathit{O}_n))$ you were asking about is surjective and two-to-one, mapping two opposite signatures $(p,q)$ and $(q,p)$ to the same real form $\mathit{O}_{p,q}$. For $n$ even, things are messier, and you should note the existence of the group $\mathit{O}^*_n$ (of maps preserving a skew-hermitian quaternionic form of quaternionic rank $n/2$; see, e.g., Onishchik & Sulanke, Projective and Cayley-Klein Geometries, Springer 2006, §2.1.6) which is a real form of $\mathit{O}_n$ that does not come from a real quadratic form; I think $H^1(\mathbb{R}, \mathrm{Aut}(\mathit{O}_n))$ has cardinality exactly $\frac{n}{2}+1$ and that the map $H^1(\mathbb{R},\mathit{O}_n) \to H^1(\mathbb{R}, \mathrm{Aut}(\mathit{O}_n))$ is two-to-one everywhere except over $\mathit{O}_{k,k}$ where $k=n/2$ and except that $\mathit{O}^*_n$ is not in its image, but I am not sure.