Groups of matrices that preserve several quadratic forms

Question

Given two (or more) quadratic forms (on the same vector space) consider the group of matrices that preserve these forms, i.e. $Q_i=U Q_i U^T$, $i=1,2..,k$ What is known about such groups? (at least for k=2 and the forms are symmetric and one is of full rank) The keywords? Where to read?

Answer 1

Let $V$ be the vector space you begin with. As you probably know, transformations $T \in \operatorname{End}(V)$ that preserve a symmetric form $(-,-)$ of full rank are called orthogonal, and the group of these transformations is denoted $O(V)$ (let me work with transformations instead of matrices).

Now, given any other form $\langle -,- \rangle$, there must exist a transformation $A \in \operatorname{End}(V)$ such that $\langle v,w \rangle = (Av, w)$, since $(-,-)$ had full rank. More precisely, one can view forms as linear transformations $V \to V^*$ via the map $v \mapsto (v, -)$ and similarly $v \mapsto \langle v, - \rangle$, and full rank ones are invertible, so we can obtain $A$ by composing $\langle -,- \rangle$ with the inverse of $(-,-)$. This obtains the desired transformation $A$.

Thus you are asking for the subgroup of $O(V)$ which also preserves $\langle v, w \rangle = (Av, w)$. This is nothing but the subgroup of $O(V)$ of transformations which commute with $A$. Indeed, if $B$ is orthogonal, then $\langle Bv, Bw \rangle = (ABv, Bw) = (B^{-1} AB v, w)$, which equals $\langle v,w \rangle$ for all $v$ and $w$ if and only if $A=B^{-1}AB$.

Of course this generalizes to the setting where you have your original nondegenerate symmetric form and $k$ other forms $v,w \mapsto (A_i v, w)$: then you are interested in the subgroup of $O(V)$ of transformations commuting with all $A_i$.

Computing this group is a standard exercise in linear algebra. As pointed out by the next author, returning to the case where $k=1$ and $A=A_1$, one can restrict to the generalized eigenspaces of $A$, which are each preserved by all $B$ in the desired group, and ask that $B$ commute with $A$ on each of those. For example, if you are working over the field of real numbers and your first form is an inner product (i.e., positive-definite), and the transformation $A$ is diagonalizable over the complex numbers (i.e., the generalized eigenspaces are all actual eigenspaces), then, up to conjugation, your group is a direct product of $O(V_\lambda)$ for the real eigenspaces $V_\lambda$ along with $U(V_{\lambda,\bar \lambda})$ for the complex nonreal pairs of eigenvalues $\lambda, \bar \lambda$ (where $V_{\lambda, \bar \lambda} \subseteq V$ has the property that its complexification is the sum of complex eigenspaces of $\lambda$ and $\bar \lambda$, and the group $U(V_{\lambda,\bar \lambda})$ is the unitary group of $V_{\lambda, \bar \lambda}$ equipped with a complex structure given by $A$, which makes the original inner product into a Hermitian one with respect to this complex structure).

Answer 2

A starting point is that such a $U$ commuttes with $Q_2Q_1^{-1}$. This reduces the analysis to the restriction of $U$ to generalized eigenspaces of $Q_2Q_1^{-1}$.

In particular, suppose that $Q_1$ (or some linear combination of the $Q_j$'s) is positive definite. Then one may choose a basis in which $Q_1=I_n$ and $Q_2$ is diagonal. Then $U$ is block diagonal, with each block being associated with one eigenvalue of $Q_2$. In general, the group is very small.