# Stabilization of the pencil of skew symmetric matrices by the orthogonal group

During my researches I've come across the following question.

Let $A$ and $B$ be a couple of square $k\times k$ skew symmetric matrices on $\mathbb R$. Let us consider the (real) pencil generated by $A$ and $B$ that is $$P(A,B)\doteq \lambda A+\eta B,\; \lambda,\eta\in\mathbb R.$$

What I am studying is the stabilization of $P(A,B)$ under the action of the orthogonal group $O(k)$, that is I would like to know what can be deduced about $A$ and $B$ under the following request that is

$$(*)\quad M P(A,B) M^{T}\subset P(A,B),\quad \forall M\in O(k).$$

Another interesting question for me would be to characterize all the $M\in O(k)$ such that $(*)$ holds.

What I would like to know is if there are already similar results in literature and what is known on the subject, references are warmly welcomed and I thank you all in advance for your kind help.

Best Wishes.

• Similarly as Raziel reinterpreted your first question into an exercise in representation theory, your second question is equivalent to this: What is the stabilizer of a point in $\mathrm{Gr}_2(\mathfrak{so}(k))$ (the Grassmanian of two-planes in $\mathfrak{so}(k)$) under the adjoint action of $\mathrm{O}(k)$? Or maybe rather: What are possible orbits of $\mathrm{O}(k)$ on $\mathrm{Gr}_2(\mathfrak{so}(k))$? I think this question is much harder. – Vít Tuček Apr 24 '14 at 16:00

## 2 Answers

Your first question

"what can be deduced about $A$ and $B$ under the request $(*)$"

can be reformulated as follows.

In fact, you are asking if there exists a $2$-dimensional vector subspace $V \subset \mathfrak{so}(k)$ stable under the adjoint action of $O(k)$

$$\mathrm{Ad} : O(K) \to \mathrm{Aut}(\mathfrak{so}(k))$$

defined by $\mathrm{Ad}(M)N := MNM^*$ for all $M \in O(k)$ and $N \in \mathfrak{so}(k)$

Thus you are asking if the adjoint representation of $O(k)$ is reducible (and admits a reduction of dimension $2$).

Your second question, i.e.

"characterize all the $M \in O(k)$ such that $(*)$ holds"

is indeed harder, but it has an explicit solution for $k=3$. In this case $\mathfrak{so}(3) \simeq \mathbb{R}^3$ and the adjoint representation of $O(3)$ is just the tautological representation.

Under the above identification, the stabilizer of a two dimensional subspace in $\mathfrak{so}(3)$ under the adjoint action is just the stabilizer of a two dimensional subspace in $\mathbb{R}^3$ under the tautological representations, namely the subgroup $SO(2)$.

• Useful rephrasing. – J. Martel Apr 24 '14 at 15:28
• Indeed, the adjoint representation of the orthogonal group is irreducible which means that the only invariant subspaces are trivial representation $0$ and the whole space $\mathfrak{so}(k)$. – Vít Tuček Apr 24 '14 at 15:39
• This settles the first question. The second one sounds harder to me. – Raziel Apr 24 '14 at 15:41
• I should probably add that the adjoin representation is not irreducible for $k=4$ where $\mathfrak{so}(4) = \mathfrak{sl}(2)\oplus\mathfrak{sl}(2)$. – Vít Tuček Apr 24 '14 at 15:45
• The isomorphism on the group level is given here mathoverflow.net/a/81359/6818 in the real version and here terrytao.wordpress.com/2011/03/11/… in the complex version. – Vít Tuček Apr 24 '14 at 15:54

It just occurred to me that the answer to your second question might be contained in the existing literature. I will give a separate answer to give it more visibility and also because is different, in spirit, from the previous one.

Normal forms for pairs of skew-symmetric matrices are known: Pencils of complex and real symmetric and skew matrices or also Canonical forms for symmetric/skew-symmetric real matrix pairs under strict equivalence and congruence

So, once the normal forms are given, it remains to compute the subgroup of $M \in O(k)$ that preserves these normal forms. This will give (possibly after some heavy and/or non-trivial linear algebra) a complete answer to your problem, which will depend indeed on the invariants of the pair.