Derivation on $SO(3)$ Let
$$u:\mathbb{R}\ni t \mapsto u(t)\in\mathcal{S}, \quad v:\mathbb{R}\ni t \mapsto v(t)\in\mathcal{S}$$
where $\mathcal{S}$ is the unit sphere of $\mathbb{R}^3$.
Consider
\begin{align} R:\ \mathcal{S}^2 &\to SO(3)  \\\  (u,v) & \mapsto (u\cdot v)I_3 + \operatorname{hat}(u\times v) + \dfrac{(u\times v) \otimes (u\times v)}{1+u\cdot v} \end{align}
with $\operatorname{hat}$ the hat operator. $R\in SO(3)$ so $R^\top \dot R$ is an antisymmetric matrix and there exists a $\omega$ such that $R^\top \dot R = \hat{\omega}$ (note: $\dot R$ is $\frac{dR}{dt}$).
After much effort and many steps I eventually found out with the help of symbolic computation program that $\omega$ simply reduces to:
$$\omega = \dfrac{-1}{1+u\cdot v} \Big((\operatorname{hat}(u+v))\dot u + (-\operatorname{hat}(u+v)+2uw^\top)\dot v\Big)$$
with $w=u\times v$.
Question Is there a simple way of finding such a simple formula?
 A: EDIT: With Willie's comment below, I had to modify my answer, and so far it is really only a partial one. I'm not quite sure how to neatly evaluate the integral of the matrix exponential. I think this should be doable, I'm just not quite sure how. Perhaps it still provides some insight. However, luckily Willie posted a more fruitful approach as an answer!

Assuming $u$ and $v$ are linearly independent, we find that the matrix $R$ fulfils
$$R(u \times v) = (u \cdot v) u \times v + \frac{1}{1 + u \cdot v} \|u \times v\|^2 u \times v \\= \left(\cos \theta + \frac{1}{1 + \cos \theta} \sin^2 \theta \right) u \times v = u \times v,
$$
so $u \times v$ lies along the rotational axis of $R$. We also have
$$u \cdot Ru = u \cdot v = \cos \theta,$$
where $\theta$ is the angle between $u$ and $v$. Since $u$ is perpendicular to the rotational axis of $R$, the angle by which $R$ rotates is then exactly $\theta$.
Because of this, if we normalize $\Omega := \frac{u \times v}{\|u \times v\|}$, then $R = \exp( \theta \cdot \hat \Omega)$ as a general property of rotational matrices, see for example this discussion related to Rodriguez' formula.
Further, we now have the differentiation formula (mentioned by Willie in the comments):
$$R^T \dot{R} = 
\exp(-\theta \hat \Omega) \cdot \int_0^1 \exp((1 - \alpha) \theta \hat \Omega) \cdot \frac{d}{dt} (\theta \cdot \hat \Omega) \exp(\alpha \theta \hat \Omega) d \alpha 
\\= 
\int_0^1 \exp(- \alpha \theta \hat \Omega) \cdot \frac{d}{dt} (\theta \cdot \hat \Omega) \exp(\alpha \theta \hat \Omega) d \alpha
\\
= \text{hat} \left( \int_0^1 \exp(- \alpha \theta \hat \Omega) \cdot \frac{d}{dt} (\theta \cdot \Omega) d \alpha \right).
$$
The last equation follows since the hat operator fulfils $ R \hat{x} R^{-1} = \widehat{R \cdot x}$ for rotation matrices $R$, and by linearity of the hat operator, which implies that it commutes with scalar multiplication and integrals.
Hence, your $\omega$ equals $\int_0^1 \exp(- \alpha \theta \hat \Omega) \cdot \frac{d}{dt} (\theta \cdot \Omega) d \alpha$. We can simplify this further: $\exp(- \alpha \theta \hat \Omega)$ still equals a rotation around the axis $\Omega$, hence $\Omega$ is an eigenvector:
$$\int_0^1 \exp(- \alpha \theta \hat \Omega) \cdot \frac{d}{dt} (\theta \cdot \Omega) d \alpha
=
\dot \theta \cdot \Omega + \int_0^1 \exp(- \alpha \theta \hat \Omega) \cdot \theta \dot{\Omega} d \alpha
$$
The evaluation of the integral of the matrix exponential can now be performed somewhat nicely using the methods from here, but I find it difficult to phrase it a super neat way here.
After that, the only thing left to do is to calculate the derivatives of $\theta = \arccos (u \cdot v)$ and $\Omega = \frac{ u \times v}{\|u \times v\|}$. Still some work to do, but this will probably give you a formula similar to yours or Willie's.
A: Let me work out the case where $\dot{v} = 0$. The case $\dot{u} = 0$ is similar and you can add using linearity of the derivative mapping.
The matrix $R(u,v)$ is the rotation of the vector $u$ to the vector $v$, in the plane spanned by $u$ and $v$. For our purposes it is convenient to define
$$ e_1 = \frac{u - (u\cdot v) v}{\| u - (u\cdot v) v\|}, \quad e_2 = v, \quad e_3 = e_1 \times e_2 = \frac{u \times v}{\|u \times v\|} $$
For small $t$, we can write $u(t) = \exp(\phi(t)e_1\wedge e_3)\exp( \theta(t) e_1 \wedge e_2) u$, with $\phi(0) = \theta(0) = 0$. (Imagine $v$ as the north pole, we first move on latitude and then change longitude.) Notice that
$$\dot{u}(0) = \dot{\phi}(0) (e_1 \wedge e_3) u + \dot{\theta}(0)(e_1\wedge e_2) u $$
Using the definitions of the frame we note that $e_3 \cdot u = 0$ and so
$$ \dot{\phi}(0) = - \frac{\dot{u}(0) \cdot e_3}{e_1 \cdot u} $$
Notice that $R(u,v)$ is of the form $\exp(\tilde{\theta} e_1\wedge e_2)$ for some $\tilde{\theta}$.
Thus
$$ R(u(t),v) = \exp(\phi(t) e_1 \wedge e_3)\exp((\tilde{\theta} - \theta(t))e_1\wedge e_2)\exp(-\phi(t) e_1 \wedge e_3) $$
And so
$$ R^T \dot{R}|_{t = 0} = \dot{\phi}(0) \cdot \left[ R^T (e_1 \wedge e_3) R - e_1 \wedge e_3\right] - \dot{\theta}(0) e_1 \wedge e_2$$
Since $e_3$ is fixed by $R$, we have that
$$ R^T \dot{R} = \dot{\phi}(0) (R^T e_1)\wedge e_3 - \dot{\phi}(0)(e_1 \wedge e_3) - \dot{\theta}(0) e_1 \wedge e_2 $$
Now, $R^T v = u$ and $R^T u =  - v + 2(v\cdot u) u$ is the reflection of $v$ to the other side of $u$, we find $R^T e_1 = \frac{(v\cdot u) u - v}{\| (v\cdot u) u - v\|}$.
Now, we see that $u, R^T e_1, e_3$ form another orthonormal frame.
Noting that $e_1 \wedge e_2 = R^T e_1 \wedge u$, and
$$ e_1  = \sqrt{1 - (u\cdot v)^2} u - (u\cdot v) R^T e_1 $$
we find
$$R^T \dot{R} = u \wedge \dot{u} - (1 - (u\cdot v))\frac{\dot{u}\cdot e_3}{e_1 \cdot u} (R^Te_1) \wedge e_3 $$
Using that $R^T e_1 \times e_3 = - u$, we conclude finally
$$ \widehat{R^T \dot{R}} = u \times \dot{u} + \frac{\dot{u} \cdot(u\times v)}{1 + (u\cdot v)}  u $$

To compare this result to what you found, notice that
$$ v = (u\cdot v) u + \sqrt{1 - (u\cdot v)^2} R^T e_1 $$
So $v \times \dot{u} = (u\cdot v) u \times \dot{u} + \sqrt{1 - (u\cdot v)^2} R^T e_1 \times \dot{u}$.
For the second term, since $\dot{u}\perp u$, we have that the cross product is actually proportional to $(\dot{u} \cdot e_3) u$. (Our formulas do not 100% agree, but I do not rule out making a small mistake here and there with numerical factors.)
