Suppose I have two matrices $\mathbf{A}\in\mathbb{R}^{n\times n}$ and $\mathbf{B}\in\mathbb{R}^{n\times n}$. I want to know what's the best similarity transformation between these two matrices when we restrict the transformation to be a rotation, i.e. how shall we solve $$ \min_{\mathbf{R}\in\mathcal{O}^{n\times n}} \\mathbf{R}^\top\mathbf{A}\mathbf{R} \mathbf{B}\_{\mathbf{F}}. $$ Here $\mathcal{O}^{n\times n}$ denotes the set of $n$ by $n$ orthonormal matrices. Can the solution be explicitly written in terms of $\mathbf{A}$ and $\mathbf{B}$, as in the case for Procrustes flow?
3

5$\begingroup$ Possible duplicate of Nearest matrix orthogonally similar to a given matrix $\endgroup$ – Josiah Park Jan 11 at 14:15

$\begingroup$ @JosiahPark Thank you! $\endgroup$ – Wuchen Jan 11 at 16:01