$\newcommand\SOg{\mathrm{SO}}\newcommand\sog{\mathfrak{so}}\newcommand\TT{\mathsf{T}}$ Key points to remember:

• $\sog(3)$ is the tangent space to $\SOg(3)$ at the identity element.
• $\SOg(3)$ is a Lie group, so associated to each element $g\in \SOg(3)$, there is a diffeomorphism $l_g: \SOg(3) \to \SOg(3)$ that sends $h\mapsto gh$. (left multiplication; you can also do right multiplication if you prefer).
• In particular, the induced tangent mapping $\TT l_g$ provides a vector space isomorphism $\sog(3) = \TT_{id} \SOg(3) \to \TT_{g}\SOg(3)$. And so you have a parametrization of the tangent bundle $\TT\SOg(3) \cong \SOg(3) \times \sog(3)$
• On the other hand, you can also realize $\SOg(3)$ as a set of $3\times 3$ matrices. Then at a particular $g$ (realized as a matrix), the tangent space $\TT_g\SOg(3)$ is a vector subspace of the space of $3\times 3$ matrices. But this vector subspace is in general different, depending on $g$.
• Working through the matrix multiplication process you see that the vector subspace turns out to be $g\cdot \sog(3)$.

When one writes "$\dot{R} = v$ with $v\in \sog(3)$", one is being a bit lazy, what one really means is that $\dot{R}(t)$ is the element in $\TT_{R(t)}\SOg(3)$ that corresponds to $v\in \sog(3)$ through the parametrization described above.

If you choose to realize $R$ as a curve in the space of $3\times 3$ matrices, the matrix corresponding to $\dot{R}(t)$ should be the matrix $R(t) \cdot v$, where $v$ is interpreted also as a $3\times 3$ (skew-symmetric) matrix.

The following may help you understand what is going on. Imagine making the following (somewhat absurd) choice to study the dynamics on $\mathbb{R}^3$: you choose to parametrize tangent vectors using the standard Euclidean system. But you choose to parametrize points in $\mathbb{R}^3$ using the cylindrical coordinates system.

Then given a curve $\gamma:(a,b)\to\mathbb{R}^3$, in the coordinate system you can write $\gamma = (r(t), z(t), \theta(t))$. But the velocity vector of $\gamma$, parametrized in the Euclidean system, is not given by $(\dot{r}(t), \dot{z}(t), \dot{\theta}(t))$, instead, its components should be $$ (\dot{r}(t) \cos(\theta(t)) - r(t) \sin(\theta(t)) \dot{\theta}(t), \dot{r}(t) \sin(\theta(t)) + r(t) \cos(\theta(t)) \dot{\theta}(t), \dot{z}(t) ) $$ which we can also write as $$ \begin{pmatrix} \cos(\theta(t)) & 0 & -r(t) \sin(\theta(t)) \\ \sin(\theta(t)) & 0 & r(t) \cos(\theta(t)) \\ 0 & 1 & 0 \end{pmatrix} \begin{pmatrix} \dot{r}(t) \\ \dot{z}(t) \\ \dot{\theta}(t) \end{pmatrix} $$

Essentially the same thing is happening: you are parameterizing the tangent space using $\sog(3)$ in a way that is not "directly compatible" with the choice of "coordinates" realizing $\SOg(3)$ as a group of matrices.

For references: you probably want to look for a book that introduces Lie Group theory with focus on Matrix Groups. Andrew Baker's Matrix Groups seems okay, except that the material most relevant to what I wrote above are relegated to an exercise or two at the end of Chapter 3. (On the other hand, if you work through and really understand Chapter 3, the exercise is very easy.)

$\newcommand\SOg{\mathrm{SO}}\newcommand\sog{\mathfrak{so}}\newcommand\TT{\mathsf{T}}$ Key points to remember:

• $\sog(3)$ is the tangent space to $\SOg(3)$ at the identity element.
• $\SOg(3)$ is a Lie group, so associated to each element $g\in \SOg(3)$, there is a diffeomorphism $l_g: \SOg(3) \to \SOg(3)$ that sends $h\mapsto gh$. (Left multiplication; you can also do right multiplication if you prefer.)
• In particular, the induced tangent mapping $\TT l_g$ provides a vector space isomorphism $\sog(3) = \TT_\mathrm{id} \SOg(3) \to \TT_{g}\SOg(3)$. And so you have a parametrization of the tangent bundle $\TT\SOg(3) \cong \SOg(3) \times \sog(3)$.
• On the other hand, you can also realize $\SOg(3)$ as a set of $3\times 3$ matrices. Then at a particular $g$ (realized as a matrix), the tangent space $\TT_g\SOg(3)$ is a vector subspace of the space of $3\times 3$ matrices. But this vector subspace is in general different, depending on $g$.
• Working through the matrix multiplication process you see that the vector subspace turns out to be $g\cdot \sog(3)$.

When one writes "$\dot{R} = v$ with $v\in \sog(3)$", one is being a bit lazy, what one really means is that $\dot{R}(t)$ is the element in $\TT_{R(t)}\SOg(3)$ that corresponds to $v\in \sog(3)$ through the parametrization described above.

If you choose to realize $R$ as a curve in the space of $3\times 3$ matrices, the matrix corresponding to $\dot{R}(t)$ should be the matrix $R(t) \cdot v$, where $v$ is interpreted also as a $3\times 3$ (skew-symmetric) matrix.

The following may help you understand what is going on. Imagine making the following (somewhat absurd) choice to study the dynamics on $\mathbb{R}^3$: you choose to parametrize tangent vectors using the standard Euclidean system. But you choose to parametrize points in $\mathbb{R}^3$ using the cylindrical coordinates system.

Then given a curve $\gamma:(a,b)\to\mathbb{R}^3$, in the coordinate system you can write $\gamma = (r(t), z(t), \theta(t))$. But the velocity vector of $\gamma$, parametrized in the Euclidean system, is not given by $(\dot{r}(t), \dot{z}(t), \dot{\theta}(t))$, instead, its components should be $$ (\dot{r}(t) \cos(\theta(t)) - r(t) \sin(\theta(t)) \dot{\theta}(t), \dot{r}(t) \sin(\theta(t)) + r(t) \cos(\theta(t)) \dot{\theta}(t), \dot{z}(t) ) $$ which we can also write as $$ \begin{pmatrix} \cos(\theta(t)) & 0 & -r(t) \sin(\theta(t)) \\ \sin(\theta(t)) & 0 & r(t) \cos(\theta(t)) \\ 0 & 1 & 0 \end{pmatrix} \begin{pmatrix} \dot{r}(t) \\ \dot{z}(t) \\ \dot{\theta}(t) \end{pmatrix} $$

Essentially the same thing is happening: you are parameterizing the tangent space using $\sog(3)$ in a way that is not "directly compatible" with the choice of "coordinates" realizing $\SOg(3)$ as a group of matrices.

For references: you probably want to look for a book that introduces Lie Group theory with focus on Matrix Groups. Andrew Baker's Matrix Groups seems okay, except that the material most relevant to what I wrote above is relegated to an exercise or two at the end of Chapter 3. (On the other hand, if you work through and really understand Chapter 3, the exercise is very easy.)