$\newcommand{\GLp}{\operatorname{GL}_n^+}$ $\newcommand{\SLs}{\operatorname{SL}^s}$ $\newcommand{\dist}{\operatorname{dist}}$ $\newcommand{\Sig}{\Sigma}$ $\newcommand{\id}{\text{Id}}$ $\newcommand{\SOn}{\operatorname{SO}_n}$ $\newcommand{\SOtwo}{\operatorname{SO}_2}$ $\newcommand{\GLtwo}{\operatorname{GL}_2^+}$

I am trying to find the **Euclidean distance** between the set of matrices of constant determinant and $\SOn$, i.e calculating
$$
F(s)= \min_{A \in \GLp,\det A=s} \dist^2(A,\SOn).
$$

Since the problem is $\SOn$-invariant we can effectively work with SVD; Using geometric reasoning, we can reduce the problem to diagonal matrices with **at most two distinct values** for its entries:

Indeed, denote by $\SLs$ the submanifold of matrices with determinant $s$; Let $\Sig \in \SLs$ be a closest matrix to $\SOn$. By orthogonal invariance, we can assume $\Sig$ is positive diagonal. Then its unique closest matrix in $\SOn$ is the identity. Consider the minimizing geodesic between $I,\Sig$: $$ \alpha(t) =\id+t(\Sig-\id). $$ Since a minimizing geodesic to a submanifold is orthogonal to it, we have $$\dot \alpha(1) \in (T_{\Sig}SL^{s})^{\perp}=(T_{(\sqrt[n]s)^{-1}\Sig}SL^{1})^{\perp}=\big((\sqrt[n]s)^{-1}\Sig T_{\id}SL^{1}\big)^{\perp}=\big(\Sig \text{tr}^{-1}(0)\big)^{\perp}.$$

Since $\Sig^{-1} \in \big(\Sig \text{tr}^{-1}(0)\big)^{\perp} $ is a basis for $\big(\Sig \text{tr}^{-1}(0)\big)^{\perp}$, we deduce

$$ \Sig-\id=\dot \alpha(1)=\lambda \Sig^{-1} \, \, \text{for some} \, \, \lambda \in \mathbb{R}, \, \text{i.e}$$

$$ \sigma_i-1=\frac{\lambda}{\sigma_i} \Rightarrow \sigma_i^2-\sigma_i-\lambda=0.$$ We see from the equation that if $\sigma_i$ is a solution, then so it $1-\sigma_i$, so if we denote by $a$ one root, the other must be $1-a$.

We just proved $\{\sigma_1,\dots,\sigma_n\} \subseteq \{a,1-a \}$.

Moreover, if the closest matrix $\Sigma$ does indeed have two distinct diagonal values, then they must be of the form $a,1-a$; Since both are positive, this implies $0<a<1$. So, we can assume WLOG that $a<\frac{1}{2}$.

*So, we are led to the following optimization problem:*

$$ F(s)=\min_{a \in (0,\frac{1}{2}),a^k(1-a)^{n-k}=s,0 \le k \le n, k \in \mathbb{N}} k(a-1)^2+(n-k)a^2. \tag{1}$$

I solved some special case below, but I don't see a good way to solve the general problem.

**Partial results so far:**

- By letting $k=0$ (or $k=n$) we get $F(s) \le n(\sqrt[n]s-1)^2$. This bound can always be realized by a conformal matrix.
$F$ decreases with the dimension: Denote by $F_n$ the function which corresponds to dimension $n$; By taking the last singular value to be $1$, we see that $F_{n+1} \le F_n$ for any $n$. In particular $F_{n} \le F_2$. ($F_2$ is computed explicitly below).

**Is the decrease strict?**In dimension $2$, a

*phase transition*occurs: I prove below that

$$F(s) = \begin{cases} 2(\sqrt{s}-1)^2, & \text{ if }\, s \ge \frac{1}{4} \\ 1-2s, & \text{ if }\, s \le \frac{1}{4} \end{cases}$$

In other words, for $A \in \GLtwo$,
$$
\dist^2(A,\SOtwo) \ge \begin{cases}
2(\sqrt{\det A}-1)^2, & \text{ if }\, \det A \ge \frac{1}{4} \\
1-2\det A, & \text{ if }\, \det A \le \frac{1}{4}
\end{cases}.
$$
When $\det A \ge \frac{1}{4}$ equality holds if and only if $A$ is **conformal**. When $\det A < \frac{1}{4}$ equality does **not** hold when $A$ is conformal: The closest matrices to $\SOtwo$ with a given determinant $s=\det A$ (up to compositions with elements in $\SOtwo$) are

$$ \begin{pmatrix} \frac{1}{2} + \frac{\sqrt{1-4\det A}}{2} & 0 \\\ 0 & \frac{1}{2} - \frac{\sqrt{1-4\det A}}{2} \end{pmatrix}, \begin{pmatrix} \frac{1}{2} - \frac{\sqrt{1-4\det A}}{2} & 0 \\\ 0 & \frac{1}{2} + \frac{\sqrt{1-4\det A}}{2} \end{pmatrix} $$

when $\det A < \frac{1}{4}$, and

$$ \begin{pmatrix} \sqrt{\det A} & 0 \\\ 0 & \sqrt{\det A} \end{pmatrix} $$

when $\det A \ge \frac{1}{4}$.

**Edit:**

By Tim's answer below, we know that if the minimizer is not conformal, then one value $0<a<\frac{1}{2}$ shows one time, and the other value (which is $1-a$) shows $n-1$ times. Since $$\max_{a \in (0,1)}a(1-a)^{n-1}=\frac{1}{n}(1-\frac{1}{n})^{n-1},$$ we deduce that if $s > \frac{1}{n}(1-\frac{1}{n})^{n-1}$ the minimizer is conformal (the other candidate "$a,1-a$" does not exist). Tim also showed that if $s \le (\frac{1}{2})^n$ then the minimizer is not conformal. It remains to determine what happens when $(\frac{1}{2})^n <s<\frac{1}{n}(1-\frac{1}{n})^{n-1}$.

Even in the case $s \le (\frac{1}{2})^n$, we do not have an explicit expression for the minimal value $F(s)$. Can we obtain such an expression? or an estimate? This amounts to estimating the smallest* root of the equation $a(1-a)^{n-1}=s$ (or equivalently finding the unique root in $(0,\frac{1}{n})$).

See here.

*Tim also showed that the winning root is the smallest one.

*Analysis of the case when $n$ is even and $n=2k$:*

**Claim:**

$$ \text{Let } \, \,F(s)=\min_{a,b \in \mathbb{R}^+,a^{\frac{n}{2}}b^{\frac{n}{2}}=s} \frac{n}{2} \big( (a-1)^2+(b-1)^2 \big). \tag{2}$$ Then $$F(s) \le f(s) := \begin{cases} n(\sqrt[n]s-1)^2, & \text{ if }\, s^{\frac{2}{n}} \ge \frac{1}{4} \\ \frac{n}{2}(1-2s^{\frac{2}{n}}), & \text{ if }\, s^{\frac{2}{n}} \le \frac{1}{4} \end{cases}$$

Expressing the constraint as $g(a,b)=ab-s^{\frac{2}{n}}=0$, and using Lagrange's multipliers method we see that there exist a $\lambda$ such that

$$ (2(a-1),2(b-1))=\lambda \nabla g(a,b)=\lambda(b,a)$$ so $a-1=\frac{b}{2}\lambda,b-1=\frac{a}{2}\lambda$.

Summing, we get $$ (a+b)-2=\frac{\lambda}{2}(a+b) \Rightarrow (a+b) (1-\frac{\lambda}{2})=2.$$ This implies $\lambda \neq 2$, so we divide and obtain $$ a+b=\frac{4}{2-\lambda} \Rightarrow a=\frac{4}{2-\lambda}-b. \tag{3}$$ So, $$a-1=\frac{4}{2-\lambda}-b-1=\frac{b}{2}\lambda \Rightarrow b(\frac{2+\lambda}{2})=\frac{2+\lambda}{2-\lambda} .$$

If $\lambda \neq -2$, then $b=\frac{2}{2-\lambda}$, which together with equation $(3)$ imply $a=b$.

Suppose $\lambda=-2$. Then $a=1-b$, so $s^{\frac{2}{n}}=ab=b(1-b)$. Since $a=1-b,b,s$ are positive we must have $0<b<1,0<s^{\frac{2}{n}}\le\frac{1}{4}$. (Since $\max_{0<b<1} b(1-b)=\frac{1}{4}$).

In that case, $$ \frac{n}{2} \big( (a-1)^2+(b-1)^2 \big) =\frac{n}{2} \big( b^2+(b-1)^2 \big)=\frac{n}{2} \big( 1-2b(1-b) \big)=\frac{n}{2}(1-2s^{\frac{2}{n}}).$$

Since $$\frac{n}{2}(1-2s^{\frac{2}{n}}) \le n(\sqrt[n]s-1)^2,$$ with equality holds iff $s^{\frac{2}{n}}=\frac{1}{4}$ we are done.

The conclusion for the $2$-dim case is immediate.