Is there a "formula" for the point in $\text{SO}(n)$ which is closest to a given matrix? $\newcommand{\Sig}{\Sigma}$
$\newcommand{\dist}{\operatorname{dist}}$
$\newcommand{\distSO}[1]{\dist(#1,\SO)}$
$\newcommand{\distO}[1]{\text{dist}(#1,\On)}$
$\newcommand{\tildistSO}[1]{\operatorname{dist}_{\til d}(#1,\SO)}$
$\newcommand{\SOn}{\operatorname{SO}_n}$
$\newcommand{\On}{\operatorname{O}_n}$
$\newcommand{\GLp}{\operatorname{GL}_n^+}$
$\newcommand{\GLtwo}{\operatorname{GL}_2^+}$
$\newcommand{\GLm}{\operatorname{GL}_n^-}$
$\newcommand{\sig}{\sigma}$
$\newcommand{\diag}{\operatorname{diag}}$
Let $A$ be a real $n \times n$ matrix, with negative determinant.
Suppose that the singular values of $A$ are pairwise distinct. Then, it can be proved that there exist a unique special orthogonal matrix $Q(A)$ which is closest to $A$ (w.r.t the Frobenius distance).

I want to find out if there exist a "formula" for $Q(A)$, say in terms of positive roots, inverses, matrix multiplication etc. Is there any hope for such a thing?

By a formula, I do not really mean a  "closed-form" formula. A closely related example of what I am looking for is the orthogonal polar factor of an invertible matrix:
If $A=OP$, where $O \in \On$, and $P$ is symmetric positive-definite, then $P=\sqrt{A^TA}$ (here $\sqrt{}$ is the unique symmetric positive-definite square root) and $O=O(A)=AP^{-1}=A(\sqrt{A^TA})^{-1}$. I consider this is an acceptable formula. 
(Comment: The orthogonal factor $O(A)$ is the closest orthogonal matrix to $A$).
Edit 2:
After some more thinking, I think the idea of a "reasonable formula" might be a bit hopeless: If we had any such "reasonable" formula, we could probably extend it continuously to all of $\GLm$. However, such a continuous extension does not exist:
Set $A_n=\begin{pmatrix} -1 & 0 \\\ 0 & 1+\frac{1}{n} \end{pmatrix},B_n=\begin{pmatrix} -(1+\frac{1}{n}) & 0 \\\ 0 & 1 \end{pmatrix}$.
Then $Q(A_n)=\text{Id},Q(B_n)=-\text{Id}$, while $A_n ,B_n$ both converge to $\begin{pmatrix} -1 & 0 \\\ 0 & 1 \end{pmatrix}$.
This phenomena implies that perhaps the best we can do is to find "partial expressions", as in Dap's answer.
Here is what I know:
Let $A=U\Sig V^T$ be the singular values decomposition of $A$; we can assume that $\Sig = \diag\left( \sig_1,\dots\sig_n \right)$ where $\sigma_1$ is the smallest singular value of $A$, and that $U \in \SOn,V \in \On,\det V=-1$.
Set $\Sig':=\diag\left( -\sig_1,\dots\sig_n \right)$, and rewrite
$$
 A= U\Sig V^T  = U (\Sig \diag\left( -1,1,1\dots ,1 \right)) (\diag\left( -1,1,1\dots ,1 \right) V ^T )
  =U \Sig' \tilde V^T,
$$
where $\tilde V \in \SOn$ is defined by requiring $\diag\left( -1,1,1\dots ,1 \right) V ^T=\tilde V^T$.

Then, it turns out that $Q(A)=U\tilde V^T$. 

Specifically, we have 
$$
\dist(A,\SOn)= \dist(U \Sig' \tilde V^T  ,\SOn)= \dist( \Sig'  ,\SOn)=d(\Sig'  ,\text{Id})\\=(\sig_1+1)^2 + \sum_{i=2}^n \left( \sig_i-1 \right)^2,
$$
and one can prove that $\text{Id}$ is the unique closest matrix in $\SOn$ to $\Sig'$. (It is important here that $\sigma_1$ is the smallest singular value of $A$).

Comment:
I prefer a formula which does not involve directly the singular vectors of $A$, since I want to understand how smoothly does $Q(A)$ varies with $A$. (The formula for the orthogonal factor mentioned above immediately implies that it is a smooth function of the matrix, once one knows that the positive square root is smooth). Finally, note that while $Q(A)=U\tilde V^T$, the orthogonal factor satisfies $O(A)=UV^T$).
Edit: I found out that the minimizer $Q(A)$ indeed changes smoothly;  This follows from the fact that locally, we can choose the matrices $U,V$ smoothly. However, I think that an elegant formula would still be a nice thing to have. (Even though we do not need one to establish smoothness). 
A more abstract discussion about smoothness of minimizers can be found here. 
 A: Not quite an answer, but we can compute $Q$ using a single suitably chosen real parameter $\lambda.$
For any $\lambda>0,$ for all matrices $A$ with singular values $\sigma_1\leq\sigma_2\leq\cdots\leq\sigma_n$ satisfying $0<\sigma_1<\lambda<\sigma_2,$
$$Q(A)=O(A)O(A^TA-\lambda^2 I)$$
where $I$ is the identity matrix, and $O(M)=M(\sqrt{M^TM})^{-1}$ as defined in the question.
To prove this, let $A=U\Sigma V^T$ as in the question. Then $\Sigma^2-\lambda^2I=|\Sigma^2-\lambda^2I|\operatorname{diag}(-1,1,1,\dots,1)$ where $|\Sigma^2-\lambda^2I|$ is a certain positive define diagonal matrix. This gives
$$O(A)O(A^TA-\lambda^2 I)=O(U\Sigma V^T)O(V|\Sigma^2-\lambda^2 I| \tilde{V}^T)=UV^TV\tilde{V}^T=U\tilde{V}^T.$$
If you were interested in integral formulas (still depending on $\lambda$): $Q$ of the form $Q(A)=A \cdot f(A^TA)$ where $f$ acts on the spectrum of $A^TA$ by taking $z>0$ to $z^{-1/2}$ for $z>\lambda$ and to $-z^{-1/2}$ for $z<\lambda.$ The holomorphic functional calculus then gives an integral formula for $f,$ locally in $A.$
Here are some relationships between the expressive power of various parameters.
If you happened to know the smallest two singular values you could plug in $\lambda=\sqrt{\sigma_1(A)\sigma_2(A)}.$ This is the inverse square root of the singular norm of $\bigwedge^2 A^{-1},$ which can be defined via spectral radius using $M^TM,$ and spectral radius has a sort of formula, Gelfand's formula. On the other hand, for symmetric $A$ with $\sigma_1(A)<\sigma_2(A),$ the matrix $\tfrac12(O(A)-Q(A))$ is projection onto the right singular vector corresponding to $\sigma_1(A).$
