I wrote an earlier post with the same question, but was asked to close the post. I now try again, essentially by adding a little background and pointing out a connection to an already studied problem,

Wahba's problem is the following:

$$\min_R \sum_{k=1}^K \|v_k - Rw_k\|^2$$ where $v_k$ and $w_k$ are arbitrary $3\times 1$ vectors, and $R$ is a rotation matrix (i.e., orthogonal with $\det(R)=1$). A variant of Wahba's problem is the orthogonal Procrustes problem.

I have a more complicated variant of Wahba's problem.

• I only study $K=3$.
• I have $w_k=R_{u_k}(\alpha_k)v_k$ where $R_{u_k}(\alpha_k)$ is a rotation around the axis $u_k$ with an angle $\alpha_k$.
• The axes $u_k$ are given in advance.
• I can optimize over $\alpha_k$.

Thus, my variant of Wahba's read $$J_{\min}=\min_{R, \alpha_k} \sum_{k=1}^3 \|v_k - RR_{u_k}(\alpha_k)v_k\|^2$$ where $v_k$ and $u_k$ are given. My claim is that there exist non-trivial solutions, i.e., $R\neq I$ and $\alpha_k\neq 0$ so that the minimum value above is $J_{\min}=0$. Note that for $R=I, \alpha_k=0$, we trivially obtain a minumum value of 0. To be more precise, what I observed in matlab is that if I choose $v_k$ and $u_k$ randomly, I always have multiple solutions achieving $J_{\min}=0$.

The background of this problem is that the orientation of a device cannot be uniquely determined using three measurements of incoming signal directions if said measurements are done with linear arrays; linear arrays result in ambiguity of incoming direction (the $\alpha_k$). The $v_k$ vectors would correspond to the directions to the nodes transmitting the signals. In plain language, my optimization problem reads: "I have three known directions $v_1$-$v_3$ in a first coordinate system. Then I can measure these, up to rotations around axes $u_k$, in a second coordinate system. The device where measurements are done may be rotated with a rotation $R$, and this links the two coordinate systems. Can I uniquely determine how the device is rotated?"

What I try to say is that this is not a homework problem, but industrial research with a practical application.

Based on the (partial) answer below, I generated the following figure. I found 4 solutions.