The answer to your first question is a resounding no. An example (among many) is given by $X=\mathrm{Spec} k$, $R=k[x]/(x^2)$ and $M=k$ considered as an $R$-module through the $k$-algebra homomorphism given by $x\mapsto 0$.
As for the second question, the reason is that in general there are correction terms to this formula coming from the failure of $M$ being locally projective. In fact there is a long exact sequence $$ 0\rightarrow H^0(X,\mathcal{E}\mathrm{xt}^1_R(M,M))\rightarrow \mathrm{Ext}^1_R(M,M)\rightarrow H^1(X,\mathcal{H}\mathrm{om}_R(M,M)) \rightarrow H^0(X,\mathcal{E}\mathrm{xt}^2_R(M,M)), $$$$ 0\rightarrow H^1(X,\mathcal{H}\mathrm{om}_R(M,M))\rightarrow \mathrm{Ext}^1_R(M,M)\rightarrow H^0(X,\mathcal{E}\mathrm{xt}^1_R(M,M)), $$ where the $\mathcal{E}\mathrm{xt}^i_R(M,M)$ are the sheaves of Ext-classes (their stalks at $x$ are the $\mathrm{Ext}^i_{R_x}(M_x,M_x)$). Hence, you need something like (possibly something a little bit weaker) the vanishing $\mathrm{Ext}^i_{R_x}(M_x,M_x)$ for $i=1,2$$i=1$ which in turn are implied by (though not implying) the local $R$-projectivity of $M$. This sequence is most easily obtained by the right hand map taking a sequence to the isomorphism clases of local extensions and the second map is obtained by twisting the trivial sequence by a torsor over its automorphism group. the exactness in the middle comes from the fact that any locally trivial sequence comes from such a twisting.
Addendum: Yes,Corrected a typo (stacks) and changed the exact sequence comesto be correct (one way to get it is from the local to global spectral sequence and the last map is a $d_2$-differential and really is necessaryI had, as I unfortunately do too often, flipped it 45 degrees).