$\newcommand{\Sig}{\Sigma}$ $\newcommand{\dist}{\operatorname{dist}}$ $\newcommand{\distSO}[1]{\dist(#1,\SO)}$ $\newcommand{\distO}[1]{\text{dist}(#1,\On)}$ $\newcommand{\tildistSO}[1]{\operatorname{dist}_{\til d}(#1,\SO)}$ $\newcommand{\SOn}{\operatorname{SO}_n}$ $\newcommand{\On}{\operatorname{O}_n}$ $\newcommand{\GLp}{\operatorname{GL}_n^+}$ $\newcommand{\GLtwo}{\operatorname{GL}_2^+}$ $\newcommand{\GLm}{\operatorname{GL}_n^-}$ $\newcommand{\sig}{\sigma}$ $\newcommand{\diag}{\operatorname{diag}}$

Let $A$ be a real $n \times n$ matrix, with **negative determinant.**
Suppose that the singular values of $A$ are pairwise distinct. Then, it can be proved that there exist a **unique** special orthogonal matrix $Q(A)$ which is closest to $A$ (w.r.t the Frobenius distance).

I want to find out if there exist a "formula" for $Q(A)$, say in terms of positive roots, inverses, matrix multiplication etc. Is there any hope for such a thing?

By a formula, I do not really mean a "closed-form" formula. A closely related example of what I am looking for is the orthogonal polar factor of an invertible matrix:

If $A=OP$, where $O \in \On$, and $P$ is symmetric positive-definite, then $P=\sqrt{A^TA}$ (here $\sqrt{}$ is the unique symmetric positive-definite square root) and $O=O(A)=AP^{-1}=A(\sqrt{A^TA})^{-1}$. I consider this is an acceptable formula.

**(Comment: The orthogonal factor $O(A)$ is the closest orthogonal matrix to $A$).**

**Edit 2:**

After some more thinking, I think the idea of a "reasonable formula" might be a bit hopeless: If we had any such "reasonable" formula, we could probably extend it continuously to all of $\GLm$. However, such a continuous extension does not exist:

Set $A_n=\begin{pmatrix} -1 & 0 \\\ 0 & 1+\frac{1}{n} \end{pmatrix},B_n=\begin{pmatrix} -(1+\frac{1}{n}) & 0 \\\ 0 & 1 \end{pmatrix}$.

Then $Q(A_n)=\text{Id},Q(B_n)=-\text{Id}$, while $A_n ,B_n$ both converge to $\begin{pmatrix} -1 & 0 \\\ 0 & 1 \end{pmatrix}$.

This phenomena implies that perhaps the best we can do is to find "partial expressions", as in Dap's answer.

Here is what I know:
Let $A=U\Sig V^T$ be the singular values decomposition of $A$; we can assume that $\Sig = \diag\left( \sig_1,\dots\sig_n \right)$ where $\sigma_1$ is the **smallest** singular value of $A$, and that $U \in \SOn,V \in \On,\det V=-1$.

Set $\Sig':=\diag\left( -\sig_1,\dots\sig_n \right)$, and rewrite $$ A= U\Sig V^T = U (\Sig \diag\left( -1,1,1\dots ,1 \right)) (\diag\left( -1,1,1\dots ,1 \right) V ^T ) =U \Sig' \tilde V^T, $$ where $\tilde V \in \SOn$ is defined by requiring $\diag\left( -1,1,1\dots ,1 \right) V ^T=\tilde V^T$.

Then, it turns out that $Q(A)=U\tilde V^T$.

Specifically, we have
$$
\dist(A,\SOn)= \dist(U \Sig' \tilde V^T ,\SOn)= \dist( \Sig' ,\SOn)=d(\Sig' ,\text{Id})\\=(\sig_1+1)^2 + \sum_{i=2}^n \left( \sig_i-1 \right)^2,
$$
and one can prove that $\text{Id}$ is the unique closest matrix in $\SOn$ to $\Sig'$. (It is important here that $\sigma_1$ is the **smallest** singular value of $A$).

*Comment:*

I prefer a formula which does not involve directly the **singular vectors** of $A$, since I want to understand how smoothly does $Q(A)$ varies with $A$. (The formula for the orthogonal factor mentioned above immediately implies that it is a smooth function of the matrix, once one knows that the positive square root is smooth). Finally, note that while $Q(A)=U\tilde V^T$, the orthogonal factor satisfies $O(A)=UV^T$).

**Edit:** I found out that the minimizer $Q(A)$ indeed changes smoothly; This follows from the fact that **locally**, we can choose the matrices $U,V$ smoothly. However, I think that an elegant formula would still be a nice thing to have. (Even though we do not need one to establish smoothness).

A more abstract discussion about smoothness of minimizers can be found here.