Existence of $SO(n)$-isotropic inner products which are not $O(n)$-isotropic

Question

$\newcommand{\al}{\alpha}$

Let $M_n$ be the space of $n \times n$ real matrices.

Question:

For which $n$, is there an inner product on $M_n$ which satisfies:

$$(*) \, \, \langle Q^TXQ,Q^TYQ \rangle = \langle X, Y\rangle $$

For every $Q \in SO(n)$,

but does not satisfy $(*)$ for every $Q \in O(n)$.

(i.e, is there an inner product which is $SO(n)$-isotropic but not $O(n)$-isotropic?)

Results so far:

$(1)$ Since for odd $n$, $-Id \in O(n)\setminus SO(n)$ commutes with every matrix in $M_n$, every $SO(n)$-invariant inner product is also $O(n)$-invariant. Thus the question is interesting only for even $n$.

$(2)$ A natural approach for this problem, is to use Riesz representation, as follows:

Let $\langle , \rangle_F$ denote the Frobenius inner product on $M_n$, and let $\langle , \rangle$ be some arbitrary inner product. Fixing $X \in M_n$, we get a linear functional: $Y \to \langle X, Y \rangle$. There is a (unique) matrix $\al(x)$ such that:

$$(**) \, \langle X, \cdot \rangle = \langle \al(X), \cdot \rangle_F$$

So, to every inner product $\langle , \rangle$ there is an associated linear operator $\al:M_n \to M_n$ satisfying $(**)$.

We say that $\al$ is $SO(n)$ (or $O(n)$)-isotropic if

$$ \al(Q^TXQ)=Q^T\alpha(X)Q $$ For every $Q \in SO(n)$ (or $O(n)$)

It's easy to see (using the fact $\langle , \rangle_F$ is left-and right $O(n)$ invariant), that:

$$\langle , \rangle \text{ is } SO(n)\text{-isotropic } \iff \al \text{ is } SO(n)\text{-isotropic }$$ ,

(and similarly to $O(n)$).

Hence, the question can be partially reduced to finding $SO(n)$-isotropic operators which are not $O(n)$-isotropic. If no such operator exists, then we finished. However, it could be the case that such an operator does exist, but does not give rise to an inner product* via $(**)$.

This is the case of the operator $\al(X)=R_{\theta}\cdot X$ , where $R_{\theta}$ is a rotation matrix, and $n=2$. (For details see here). The corrseponding bilinear form turns out to be positive but not symmetric. Taking its symmetrization gives an inner product, which turns out to be $O(n)$-isotropic.

I do not know if there are any such operators for even $n \neq 2$.

In fact, for $O(n)$-invariant operators, there is a representation theorem which says they must be of the form of:

$\alpha(X)=a \text{tr}(X)I + bX + cX^T$

A proof is given for example here.

---

*To induce an inner product, $\al$ must be self-adjoint w.r.t the Frobenius product, and be positive, in the sense that $\langle \al(X), X\rangle_F > 0 \, , \, \forall X \neq 0$

Answer 1

Disclaimer: this answer is at least 50% due to @Holonomia. If you like it, why don't you upvote some of her posts?

The groups $O(n)$ and $SO(n)$ act by conjugation on $M_n(\mathbb R)$. There is an equivariant decomposition $$M_n(\mathbb R)\cong\mathbb R E_n\oplus\mathrm{Sym}_0^2(\mathbb R^n) \oplus\Lambda^2(\mathbb R^n)\;.$$ The second summand consists of trace-free symmetric matrices, the third one of skew-symmetric matrices. Each of these spaces is an irreducible $O(n)$-representation (or $0$ if $n$ is small). For almost all $n$, each one is also irreducible as an $SO(n)$-representation (or $0$).

The only exception is $n=4$. Here, $\Lambda^2(\mathbb R^4)\cong\Lambda^{2,+}(\mathbb R^4)\oplus\Lambda^{2,-}(\mathbb R^4)$ as an $SO(4)$-representation. This is the decomposition into the $\pm 1$-eigenspaces (also called selfdual and antiselfdual forms) of the Hodge-$*$-operator, which needs a metrics and an orientation in its definition, and hence is $SO(4)$-invariant. Typical elements are $$e_1\wedge e_2+e_3\wedge e_4=\begin{pmatrix}&1\\-1\\&&&1\\&&-1\end{pmatrix}\in\Lambda^{2,+}(\mathbb R^4)$$ and $$e_1\wedge e_2+e_4\wedge e_3=\begin{pmatrix}&1\\-1\\&&&-1\\&&1\end{pmatrix}\in\Lambda^{2,-}(\mathbb R^4)\;.$$ Replacing the standard basis by another oriented ONB gives more elements. The elements of $O^-(4)=O(4)\setminus SO(4)$ swap both spaces. There is an $SO(4)$- but not $O(4)$-invariant scalar product on $\Lambda^2(\mathbb R^4)$ such that $$\langle\alpha,\beta\rangle\,e_1\wedge\cdots\wedge e_4=\alpha\wedge\beta\;.$$ For more details, please find a nice introduction to the representation theory of compact Lie groups.

So the only $SO(n)$- but not $O(n)$-invariant scalar products on $M_n(\mathbb R)$ exist in dimension $n=4$.

Here is an explanation: Let $G$ be any compact Lie group, and let $V$ be any $G$-representation over $\Bbbk=\mathbb R$ or $\mathbb C$. Then $V$ admits a decomposition $V\cong V_1\otimes\Bbbk^{n_1}\oplus\dots\oplus V_k\otimes\Bbbk^{n_k}$. Here $V_1,\dots,V_n$ are irreducible, and $\gamma\in G$ acts as $\gamma\otimes\mathrm{id}$ on each summand. This decomposition is unique up to reordering. Fix a $G$-invariant scalar product $g_i$ on each $V_i$. Then each $G$-invariant scalar product on $V$ is of the form $g=g_1\otimes h_1\oplus\cdots\oplus g_k\otimes h_k$, where $h_i$ is an arbitrary scalar product on $\Bbbk^{n_i}$. This is an application of Schur's lemma that you can find in most introductory books on representation theory.

If $H\subset G$ is a compact subgroup, then each irreducible $G$-representation $V_i$ above is an $H$-representation, so it has an analogous decomposition. From this it is clear that every $H$-invariant scalar product on $V$ is $G$ invariant if and only if each $V_i$ in the decomposition above is also irreducible as an $H$-representation (so the decomposition of $V_i$ over $H$ is just $V_i\cong V_i\otimes\Bbbk$).