Lets write π:R->End(M) for the representation.
Let S be a F-algebra. We first eludicate the S-points of $Aut_R(M)$. We have $$Aut_R(M)(S)=\{ g\in GL_n(S)| g\pi(r)g^{-1}=\pi(r)\forall r\in R \}.$$ We also consider the S-points of Hom(M,M). Hom(M,M) has the same functor of points, except we replace the $g\in GL_n(S)$ condition by $g\in Mat_n(S)$. Note that Hom(M,M) is an affine space, since all conditions are linear. And we read off from their functors of points that $Aut_R(M)$ is the open set in Hom(M,M) where the determinant function is invertible. So $Aut_R(M)$ indeed is open in an affine space, hence smooth.
