The answer to your first question is a resounding no. An example (among many) is
given by $X=\mathrm{Spec} k$, $R=k[x]/(x^2)$ and $M=k$ considered as an
$R$-module through the $k$-algebra homomorphism given by $x\mapsto 0$.
As for the second question, the reason is that in general there are correction
terms to this formula coming from the failure of $M$ being locally
projective. In fact there is a long exact sequence
$$
0\rightarrow H^1(X,\mathcal{H}\mathrm{om}_R(M,M))\rightarrow
\mathrm{Ext}^1_R(M,M)\rightarrow H^0(X,\mathcal{E}\mathrm{xt}^1_R(M,M)),
$$
where the $\mathcal{E}\mathrm{xt}^i_R(M,M)$ are the sheaves of Ext-classes
(their stalks at $x$ are the $\mathrm{Ext}^i_{R_x}(M_x,M_x)$). Hence, you need
something like (possibly something a little bit weaker) the vanishing
$\mathrm{Ext}^i_{R_x}(M_x,M_x)$ for $i=1$ which in turn are implied by (though not
implying) the local $R$-projectivity of $M$. This sequence is most easily obtained by the right hand map taking a sequence to the isomorphism clases of local extensions and the second map is obtained by twisting the trivial sequence by a torsor over its automorphism group. the exactness in the middle comes from the fact that any locally trivial sequence comes from such a twisting.
Addendum: Corrected a typo (stacks) and changed the exact sequence to be correct (one way to get it is from the local to global spectral sequence and I had, as I unfortunately do too often, flipped it 45 degrees).