The orthogonal Procrustes problem seeks a matrix $M$ that minimizes $||AM-B||_F$ subject to $M^TM=I$, where $M$ is $d\times d$ and both $A$ and $B$ are $n\times d$. Geometrically, $M$ rotates a set of $n$ vectors (the rows of $A$) so that they best align with another set of $n$ vectors (the rows of $B$) in a least-squares sense.
It has been shown that there is an elegant closed-form solution to this problem, namely $M=UV^T$ where $A^TB=U\Sigma V^T$.
I am interested in solving a similar problem and wondering if there is an equally elegant closed-form solution. Given vector sets $A$ and $B$ as before, I would like to find $M$ that makes the vectors in $A$ as "orthogonal" as possible to corresponding vectors in $B$. That is, I want $a_i^TMb_i$ to be as close to zero as possible for $i\in[1,n]$, where $a_i$ and $b_i$ represent particular rows of $A$ and $B$:
$M^*=\underset{M}{\operatorname{argmin}}\sum_i (a_i^TMb_i)^2$ subject to $M^TM=I$.
Is there a name for this formulation? Anyone know of a solution that can be easily expressed (and computed) as a function of $A$ and $B$, or a reason that it cannot?
