We are in $M_n(\mathbb{R})$ equipped with the Frobenius norm $||A||^2=tr(AA^T)$.
Let $Z=\{(A,B)\in M_n(\mathbb{R})^2;A^2-AB-B^2=0\}$ and $T=O(n)^2$. It is easy to see that $Z\cap T=\emptyset$ and obvious that $Z$ is a cone and $T$ a compact set..
Thus $m_n=\min_{(A,B)\in Z,(X,Y)\in T}(||A-X||^2+||B-Y||^2)>0$.
$\textbf{Question}.$ Is it true that $m_n=n(1-2/\sqrt{5})$ ?
$\textbf{Miscellaneous}$. 1. if $(A,B)\in Z$, then $A,B$ are simultaneously triangularizable (at least over $\mathbb{C})$.
- Numerical calculations for $n=2,3,4$ seem to support this conjecture; moreover this bound would be reached for $X_0=I_n,Y_0=permut(n,\cdots,1)$ and $A_0,B_0\in M_n(\mathbb{Q}\sqrt{5})$.
For example, when $n=4$, $A_0,B_0$ are
Note that the considered bound is reached in other points of $Z\times T$ whose coordinates are more complicated.
- If $A_0,B_0,X_0,Y_0$ realize $m_n$, then (for the Frobenius scalar product)
$(A_0-X_0,B_0-Y_0)$ is orthogonal to the tangent space of $Z$ in $(A_0,B_0)$ AND to the tangent space of $T$ in $(X_0,Y_0)$.
- What prompted me to ask this question is the explicit form of the conjectured solution.
Note that the "$\sqrt{5}$" term may come from the equation $A^2-AB-B^2=0$. Indeed, putting $A=uP+vQ,B=wP+tQ$ (where $u,v,w,t$ are well chosen), we can reduce the problem to $PQ=\dfrac{-3+\sqrt{5}}{2}QP$.
