Stabilization of the pencil of skew symmetric matrices by the orthogonal group

Question

During my researches I've come across the following question.

Let $A$ and $B$ be a couple of square $k\times k$ skew symmetric matrices on $\mathbb R$. Let us consider the (real) pencil generated by $A$ and $B$ that is $$P(A,B)\doteq \lambda A+\eta B,\; \lambda,\eta\in\mathbb R.$$

What I am studying is the stabilization of $P(A,B)$ under the action of the orthogonal group $O(k)$, that is I would like to know what can be deduced about $A$ and $B$ under the following request that is

$$(*)\quad M P(A,B) M^{T}\subset P(A,B),\quad \forall M\in O(k).$$

Another interesting question for me would be to characterize all the $M\in O(k)$ such that $(*)$ holds.

What I would like to know is if there are already similar results in literature and what is known on the subject, references are warmly welcomed and I thank you all in advance for your kind help.

Best Wishes.

Answer 1

Your first question

"what can be deduced about $A$ and $B$ under the request $(*)$"

can be reformulated as follows.

In fact, you are asking if there exists a $2$-dimensional vector subspace $V \subset \mathfrak{so}(k)$ stable under the adjoint action of $O(k)$

$$\mathrm{Ad} : O(K) \to \mathrm{Aut}(\mathfrak{so}(k))$$

defined by $\mathrm{Ad}(M)N := MNM^*$ for all $M \in O(k)$ and $N \in \mathfrak{so}(k)$

Thus you are asking if the adjoint representation of $O(k)$ is reducible (and admits a reduction of dimension $2$).

Your second question, i.e.

"characterize all the $M \in O(k)$ such that $(*)$ holds"

is indeed harder, but it has an explicit solution for $k=3$. In this case $\mathfrak{so}(3) \simeq \mathbb{R}^3$ and the adjoint representation of $O(3)$ is just the tautological representation.

Under the above identification, the stabilizer of a two dimensional subspace in $\mathfrak{so}(3)$ under the adjoint action is just the stabilizer of a two dimensional subspace in $\mathbb{R}^3$ under the tautological representations, namely the subgroup $SO(2)$.

Answer 2

It just occurred to me that the answer to your second question might be contained in the existing literature. I will give a separate answer to give it more visibility and also because is different, in spirit, from the previous one.

Normal forms for pairs of skew-symmetric matrices are known: Pencils of complex and real symmetric and skew matrices or also Canonical forms for symmetric/skew-symmetric real matrix pairs under strict equivalence and congruence

So, once the normal forms are given, it remains to compute the subgroup of $M \in O(k)$ that preserves these normal forms. This will give (possibly after some heavy and/or non-trivial linear algebra) a complete answer to your problem, which will depend indeed on the invariants of the pair.