Every $A \in \text{GL}_n(\mathbb{R})$ has a unique orthogonal polar factor $O_A=A(\sqrt{A^TA})^{-1}$,
( $A=O_AP_A$, $O \in \operatorname{O}_n, P \in \operatorname{Psym}_n$see Polar decomposition).
I am interseted in characterizing the set $W:=\{(A,B)\in \text{GL}_n^+ \times \text{GL}_n^+|\,O_{AB}=O_AO_B \}$. I have made some progress, but am still lacking a unifying necessary and sufficient criterion. Even the special case where $A=B$ does not seem trivial. Any advancement would be welcome.
Edit: Another natural question is: How rare is it for a pair $(A,B)$ to be in $W$? (we can take the uniform distribution on $\text{GL}_n^+ \times \text{GL}_n^+$ and ask what is the probability of a random pair to be in $W$).
Motivation:
This question has a geometric nature, since $O_A$ is the closest matrix in $\text{SO}_n$ to $A$ (w.r.t to the Frobenius norm). Thus, it reflects the "rotational part" of $A$, after the scaling of the different directions is removed; This is very clear when looking at the SVD of $A$, $$A=U\Sigma V^T \Rightarrow O_A=UV^T$$ So, we can think of $O_A$ as the best "isometric approximation" (or projection) of $A$. If we know the two approximations for $A,B$ it is natural to compose them and to ask if we obtained in this way the best approximation to $AB$. (This comes up in some applications of geometry and analysis).
What I have so far:
(0) The relation is not symmetric, i.e $(A,B) \in W \nRightarrow (B,A) \in W$.
(1) A sufficient condition for $(A,B) \in W$ is that $P_A$ commutes with $O_B,P_B$. (This is far from being a necessary condition, see below).
(2) $\forall A \in \text{GL}_n^+ \,: \, (A,A^{-1}) \in W$.
(3) If $B \in \operatorname{Psym}_n$, then $$(A,B) \in W \iff [P_A,B]=[P_A,P_B] =0$$ (see here for details )
In particular, for $A,B \in \operatorname{Psym}_n$: $$(A,B) \in W \iff [A,B]=0$$
(4) None of these implications hold in the general (proofs at the end):
(a) $(A,B) \in W \nRightarrow [A,B]=0.$
(b) $[A,B]=0 \nRightarrow (A,B) \in W.$
(c) $(A,B) \in W \nRightarrow [P_A,P_B]=0.$
(d) $[P_A,P_B]=0 \nRightarrow (A,B) \in W.$
(5) If $A$ is a normal matrix, then $(A,A) \in W$.
(This is easy to see since $A$ is normal $\iff$ $[O_A,P_A]=0$, via the uniqueness of the positive square root).
Is it true that $(A,A) \in W$ implies $A$ is normal ? (I suspect the answer is negative)
Proofs (for the statements in (4)):
(a) Take $B \in \operatorname{Psym}_n,O \in \operatorname{SO}_n$ such that $[B,O] \neq 0$, and define $A=OB$. Then $(A,B) \in W$ but $\, AB=OB^2 \neq BOB=BA$.
(b) Assume by contradiction $[A,B]=0 \Rightarrow (A,B) \in W$. Then $[A,B]=0 \Rightarrow [O_A,O_B]=0$; Indeed, $$[A,B]=0 \Rightarrow [B,A] =0 \Rightarrow (B,A) \in W,$$ so $$O_BO_A=O_{BA}=O_{AB}=O_AO_B \Rightarrow [O_A,O_B]=0.$$ However, this does not always hold:
Let $A \in \text{GL}_2^+,B \in \text{GL}_2^-$ be commuting matrices. Note that $[O_A,O_B] \neq 0$ since for any two elements $X \in \operatorname{O}_2\setminus \operatorname{SO}_2,Y \in \operatorname{SO}_2\setminus\{\pm Id\}$, $[X,Y] \neq 0$. However, I want counter-examples where $A,B$ are both in $\text{GL}_n^+$. This is impossible to find for $n=2$ since $\operatorname{SO}_2$ is abelian. However, we can take $4 \times 4$ matrices, by "doubling" $A,B$ in two diagonal blocks.
(For a concrete example see here)
(c) See counter-example here.
(d) See counter-example here.
