I apologize if this is a well known problem and I've just missed the answer, but I have searched fairly extensively. Perhaps there is another way to phrase it that I've been missing?
This question was posted four days ago at Math.SE, but with no answer or comments.
Suppose there are two submanifolds $M_1$ and $M_2$ embedded within $SO(3)$.
$M_1 =\left\{ P_1 e^{\, \mathbf \xi_1 t} | t \in \left[0, ^{2 \pi}/_{|\xi_1|} \right] \right\}$
$M_2 =\left\{ P_2 e^{\, \mathbf \xi_2 t} | t \in \left[0, ^{2 \pi}/_{|\xi_2|} \right] \right\}$
with $P_1, P_2 \in SO(3)$ and $\mathbf \xi_1, \mathbf \xi_2 \in \mathfrak{so(3)}$
Is there an analytic way to determine the pair of points on $M_1$ and $M_2$ that minimizes the geodesic distance between them.
$\arg \min d(m_1, m_2)= \| \log m_1 m_2^\top\|\; : \; m_1 \in M_1, m_2 \in M_2$
The specific problem I'm trying to solve is for a rigid body with an initial attitude and a fixed rotation rate, what is the nearest attitude it will reach with a defined roll and pitch so $\xi_1$ would also point only in the direction of $i_z$ if that further constraint helps. I can solve this numerically, but am looking for an analytic solution to try and implement this on a resource constrained platform.