Given two (or more) quadratic forms (on the same vector space) consider the group of matrices that preserve these forms, i.e. $Q_i=U Q_i U^T$, $i=1,2..,k$ What is known about such groups? (at least for k=2 and the forms are symmetric and one is of full rank) The keywords? Where to read?

$\begingroup$ Nice question! This is pretty obvious, but I point out that classifying pairs of metrics, one symmetric and one skewsymmetric, would be equivalent to classifying arbitrary 2tensors. $\endgroup$– José NavarroJul 6, 2012 at 9:49

$\begingroup$ I would write this as a comment, but this festure is unavailible to me. You may find this paper relevant/intresting. $\endgroup$– OrbitJul 6, 2012 at 12:08
2 Answers
Let $V$ be the vector space you begin with. As you probably know, transformations $T \in \operatorname{End}(V)$ that preserve a symmetric form $(,)$ of full rank are called orthogonal, and the group of these transformations is denoted $O(V)$ (let me work with transformations instead of matrices).
Now, given any other form $\langle , \rangle$, there must exist a transformation $A \in \operatorname{End}(V)$ such that $\langle v,w \rangle = (Av, w)$, since $(,)$ had full rank. More precisely, one can view forms as linear transformations $V \to V^*$ via the map $v \mapsto (v, )$ and similarly $v \mapsto \langle v,  \rangle$, and full rank ones are invertible, so we can obtain $A$ by composing $\langle , \rangle$ with the inverse of $(,)$. This obtains the desired transformation $A$.
Thus you are asking for the subgroup of $O(V)$ which also preserves $\langle v, w \rangle = (Av, w)$. This is nothing but the subgroup of $O(V)$ of transformations which commute with $A$. Indeed, if $B$ is orthogonal, then $\langle Bv, Bw \rangle = (ABv, Bw) = (B^{1} AB v, w)$, which equals $\langle v,w \rangle$ for all $v$ and $w$ if and only if $A=B^{1}AB$.
Of course this generalizes to the setting where you have your original nondegenerate symmetric form and $k$ other forms $v,w \mapsto (A_i v, w)$: then you are interested in the subgroup of $O(V)$ of transformations commuting with all $A_i$.
Computing this group is a standard exercise in linear algebra. As pointed out by the next author, returning to the case where $k=1$ and $A=A_1$, one can restrict to the generalized eigenspaces of $A$, which are each preserved by all $B$ in the desired group, and ask that $B$ commute with $A$ on each of those. For example, if you are working over the field of real numbers and your first form is an inner product (i.e., positivedefinite), and the transformation $A$ is diagonalizable over the complex numbers (i.e., the generalized eigenspaces are all actual eigenspaces), then, up to conjugation, your group is a direct product of $O(V_\lambda)$ for the real eigenspaces $V_\lambda$ along with $U(V_{\lambda,\bar \lambda})$ for the complex nonreal pairs of eigenvalues $\lambda, \bar \lambda$ (where $V_{\lambda, \bar \lambda} \subseteq V$ has the property that its complexification is the sum of complex eigenspaces of $\lambda$ and $\bar \lambda$, and the group $U(V_{\lambda,\bar \lambda})$ is the unitary group of $V_{\lambda, \bar \lambda}$ equipped with a complex structure given by $A$, which makes the original inner product into a Hermitian one with respect to this complex structure).
A starting point is that such a $U$ commuttes with $Q_2Q_1^{1}$. This reduces the analysis to the restriction of $U$ to generalized eigenspaces of $Q_2Q_1^{1}$.
In particular, suppose that $Q_1$ (or some linear combination of the $Q_j$'s) is positive definite. Then one may choose a basis in which $Q_1=I_n$ and $Q_2$ is diagonal. Then $U$ is block diagonal, with each block being associated with one eigenvalue of $Q_2$. In general, the group is very small.

$\begingroup$ When you say "positive definite" I assume for this you need the field to be a subfield of the real numbers, since otherwise this notion is not welldefined for quadratic forms (which is the context the poster used). One could also generalize positivedefiniteness to subfields of the complex numbers if one replaces quadratic forms by Hermitian forms. $\endgroup$ Jul 6, 2012 at 14:45