Is there a "formula" for the point in $\text{SO}(n)$ which is closest to a given matrix?

Question

$\newcommand{\Sig}{\Sigma}$ $\newcommand{\dist}{\operatorname{dist}}$ $\newcommand{\distSO}[1]{\dist(#1,\SO)}$ $\newcommand{\distO}[1]{\text{dist}(#1,\On)}$ $\newcommand{\tildistSO}[1]{\operatorname{dist}_{\til d}(#1,\SO)}$ $\newcommand{\SOn}{\operatorname{SO}_n}$ $\newcommand{\On}{\operatorname{O}_n}$ $\newcommand{\GLp}{\operatorname{GL}_n^+}$ $\newcommand{\GLtwo}{\operatorname{GL}_2^+}$ $\newcommand{\GLm}{\operatorname{GL}_n^-}$ $\newcommand{\sig}{\sigma}$ $\newcommand{\diag}{\operatorname{diag}}$

Let $A$ be a real $n \times n$ matrix, with negative determinant. Suppose that the singular values of $A$ are pairwise distinct. Then, it can be proved that there exist a unique special orthogonal matrix $Q(A)$ which is closest to $A$ (w.r.t the Frobenius distance).

I want to find out if there exist a "formula" for $Q(A)$, say in terms of positive roots, inverses, matrix multiplication etc. Is there any hope for such a thing?

By a formula, I do not really mean a "closed-form" formula. A closely related example of what I am looking for is the orthogonal polar factor of an invertible matrix:

If $A=OP$, where $O \in \On$, and $P$ is symmetric positive-definite, then $P=\sqrt{A^TA}$ (here $\sqrt{}$ is the unique symmetric positive-definite square root) and $O=O(A)=AP^{-1}=A(\sqrt{A^TA})^{-1}$. I consider this is an acceptable formula.

(Comment: The orthogonal factor $O(A)$ is the closest orthogonal matrix to $A$).

Edit 2:

After some more thinking, I think the idea of a "reasonable formula" might be a bit hopeless: If we had any such "reasonable" formula, we could probably extend it continuously to all of $\GLm$. However, such a continuous extension does not exist:

Set $A_n=\begin{pmatrix} -1 & 0 \\\ 0 & 1+\frac{1}{n} \end{pmatrix},B_n=\begin{pmatrix} -(1+\frac{1}{n}) & 0 \\\ 0 & 1 \end{pmatrix}$.

Then $Q(A_n)=\text{Id},Q(B_n)=-\text{Id}$, while $A_n ,B_n$ both converge to $\begin{pmatrix} -1 & 0 \\\ 0 & 1 \end{pmatrix}$.

This phenomena implies that perhaps the best we can do is to find "partial expressions", as in Dap's answer.

Here is what I know: Let $A=U\Sig V^T$ be the singular values decomposition of $A$; we can assume that $\Sig = \diag\left( \sig_1,\dots\sig_n \right)$ where $\sigma_1$ is the smallest singular value of $A$, and that $U \in \SOn,V \in \On,\det V=-1$.

Set $\Sig':=\diag\left( -\sig_1,\dots\sig_n \right)$, and rewrite $$ A= U\Sig V^T = U (\Sig \diag\left( -1,1,1\dots ,1 \right)) (\diag\left( -1,1,1\dots ,1 \right) V ^T ) =U \Sig' \tilde V^T, $$ where $\tilde V \in \SOn$ is defined by requiring $\diag\left( -1,1,1\dots ,1 \right) V ^T=\tilde V^T$.

Then, it turns out that $Q(A)=U\tilde V^T$.

Specifically, we have $$ \dist(A,\SOn)= \dist(U \Sig' \tilde V^T ,\SOn)= \dist( \Sig' ,\SOn)=d(\Sig' ,\text{Id})\\=(\sig_1+1)^2 + \sum_{i=2}^n \left( \sig_i-1 \right)^2, $$ and one can prove that $\text{Id}$ is the unique closest matrix in $\SOn$ to $\Sig'$. (It is important here that $\sigma_1$ is the smallest singular value of $A$).

---

Comment:

I prefer a formula which does not involve directly the singular vectors of $A$, since I want to understand how smoothly does $Q(A)$ varies with $A$. (The formula for the orthogonal factor mentioned above immediately implies that it is a smooth function of the matrix, once one knows that the positive square root is smooth). Finally, note that while $Q(A)=U\tilde V^T$, the orthogonal factor satisfies $O(A)=UV^T$).

Edit: I found out that the minimizer $Q(A)$ indeed changes smoothly; This follows from the fact that locally, we can choose the matrices $U,V$ smoothly. However, I think that an elegant formula would still be a nice thing to have. (Even though we do not need one to establish smoothness).

A more abstract discussion about smoothness of minimizers can be found here.

Answer 1

Not quite an answer, but we can compute $Q$ using a single suitably chosen real parameter $\lambda.$ For any $\lambda>0,$ for all matrices $A$ with singular values $\sigma_1\leq\sigma_2\leq\cdots\leq\sigma_n$ satisfying $0<\sigma_1<\lambda<\sigma_2,$ $$Q(A)=O(A)O(A^TA-\lambda^2 I)$$

where $I$ is the identity matrix, and $O(M)=M(\sqrt{M^TM})^{-1}$ as defined in the question. To prove this, let $A=U\Sigma V^T$ as in the question. Then $\Sigma^2-\lambda^2I=|\Sigma^2-\lambda^2I|\operatorname{diag}(-1,1,1,\dots,1)$ where $|\Sigma^2-\lambda^2I|$ is a certain positive define diagonal matrix. This gives

$$O(A)O(A^TA-\lambda^2 I)=O(U\Sigma V^T)O(V|\Sigma^2-\lambda^2 I| \tilde{V}^T)=UV^TV\tilde{V}^T=U\tilde{V}^T.$$

If you were interested in integral formulas (still depending on $\lambda$): $Q$ of the form $Q(A)=A \cdot f(A^TA)$ where $f$ acts on the spectrum of $A^TA$ by taking $z>0$ to $z^{-1/2}$ for $z>\lambda$ and to $-z^{-1/2}$ for $z<\lambda.$ The holomorphic functional calculus then gives an integral formula for $f,$ locally in $A.$

Here are some relationships between the expressive power of various parameters. If you happened to know the smallest two singular values you could plug in $\lambda=\sqrt{\sigma_1(A)\sigma_2(A)}.$ This is the inverse square root of the singular norm of $\bigwedge^2 A^{-1},$ which can be defined via spectral radius using $M^TM,$ and spectral radius has a sort of formula, Gelfand's formula. On the other hand, for symmetric $A$ with $\sigma_1(A)<\sigma_2(A),$ the matrix $\tfrac12(O(A)-Q(A))$ is projection onto the right singular vector corresponding to $\sigma_1(A).$