$\newcommand{\al}{\alpha}$

Let $M_n$ be the space of $n \times n$ real matrices.

**Question:**

For which $n$, is there an inner product on $M_n$ which satisfies:

$$(*) \, \, \langle Q^TXQ,Q^TYQ \rangle = \langle X, Y\rangle $$

For every $Q \in SO(n)$,

but does **not** satisfy $(*)$ for every $Q \in O(n)$.

(i.e, is there an inner product which is $SO(n)$-**isotropic** but not $O(n)$-**isotropic**?)

**Results so far:**

$(1)$ Since for odd $n$, $-Id \in O(n)\setminus SO(n)$ commutes with every matrix in $M_n$, every $SO(n)$-invariant inner product is also $O(n)$-invariant. Thus the question is interesting only for even $n$.

$(2)$ A natural approach for this problem, is to use Riesz representation, as follows:

Let $\langle , \rangle_F$ denote the Frobenius inner product on $M_n$, and let $\langle , \rangle$ be some arbitrary inner product. Fixing $X \in M_n$, we get a linear functional: $Y \to \langle X, Y \rangle$. There is a (unique) matrix $\al(x)$ such that:

$$(**) \, \langle X, \cdot \rangle = \langle \al(X), \cdot \rangle_F$$

So, to every inner product $\langle , \rangle$ there is an associated linear operator $\al:M_n \to M_n$ satisfying $(**)$.

We say that $\al$ is $SO(n)$ (or $O(n)$)-**isotropic** if

$$ \al(Q^TXQ)=Q^T\alpha(X)Q $$ For every $Q \in SO(n)$ (or $O(n)$)

It's easy to see (using the fact $\langle , \rangle_F$ is left-and right $O(n)$ invariant), that:

$$\langle , \rangle \text{ is } SO(n)\text{-isotropic } \iff \al \text{ is } SO(n)\text{-isotropic }$$ ,

(and similarly to $O(n)$).

Hence, the question can be partially reduced to finding $SO(n)$-isotropic operators which are not $O(n)$-isotropic. If no such operator exists, then we finished. However, it could be the case that such an operator does exist, but does not give rise to an inner product* via $(**)$.

This is the case of the operator $\al(X)=R_{\theta}\cdot X$ , where $R_{\theta}$ is a rotation matrix, and $n=2$. (For details see here). The corrseponding bilinear form turns out to be positive but not symmetric. Taking its symmetrization gives an inner product, which turns out to be $O(n)$-isotropic.

I do not know if there are any such operators for even $n \neq 2$.

In fact, for $O(n)$-invariant operators, there is a representation theorem which says they must be of the form of:

$\alpha(X)=a \text{tr}(X)I + bX + cX^T$

A proof is given for example here.

*To induce an inner product, $\al$ must be self-adjoint w.r.t the Frobenius product, and be positive, in the sense that $\langle \al(X), X\rangle_F > 0 \, , \, \forall X \neq 0$