Question

Given a 3 by 3 matrix $M$ I would like to find the rotation matrix $R$ minimizing the Frobenius norm:

\begin{equation} \|R-M\|_F \end{equation}

Is there a closed form solution for $R$, or is it possible to express $R$ as the solution to a linear system? I would like to avoid gradient descent if possible.

Answer 1

Let $M=U\Sigma V$ be the singular value decomposition of $M$, then $R=UV$. If you want $R$ to be a proper rotation (i.e. $\det R=1$) and $UV$ is not, replace the singular vector $\mathbf{u}_3$ associated with the smallest singular value of $M$ with $-\mathbf{u}_3$ in the $U$ matrix. The appropriate reference for this answer is http://www.ma.man.ac.uk/~nareports/narep161.pdf.