The question is too broad in general, but I believe that there is a nice answer for your particular example. The first thing to note is that the given $M$ is a twisted bimodule. Namely, let $\sigma$ be the automorphism of $R$ given by $$\sigma : \begin{pmatrix} r & s \\ xt & u \end{pmatrix} \mapsto \begin{pmatrix} u &t \\ xs &r \end{pmatrix}.$$
Then we can define a bimodule isomorphism $\varphi : {}_1 R_{\sigma} \rightarrow M$ by
$$\varphi : \begin{pmatrix} a &b \\ c &d \end{pmatrix} \mapsto \begin{pmatrix} b & x^{-1}a \\ d & c \end{pmatrix}.$$
It follows that tensoring with $M$ can be thought of as the endofunctor on the category of (left) R-modules that is induced by the automorphism $\sigma^{-1}$ ($= \sigma$). Consequently, the modules $F$ and $M \otimes_R F$ have the same structure (in a sense), but with the opposite composition factors.
(To contrast with the general situation, the assumption that tensoring with $M$ swaps the simples, alone, does not guarantee that tensoring with $M$ preserves the length of finite-length modules.)
Thus, if you know the structure of $F$, it should not be hard to see if there are any nonzero morphisms from $F$ to $M \otimes_R F$. For example, if $F$ contains a factor of $S_1$ in its top and a factor of $S_2$ in its socle, then $M \otimes_R F$ contains a factor of $S_1$ in its socle, and there will be a nonzero map from $F$ to $M \otimes_R F$. On the other hand, if $F$ is uniserial with composition factors $S_1, S_2, S_1, S_1$ (from the top down), then $\mbox{Hom}_R(F,M\otimes_R F) = 0$.
However, for a finite-length module $F$ over the given $R$, it appears plausible that $\mbox{Hom}_R(F, M \otimes_R F)=0$ if and only if all composition factors of $F$ are isomorphic. I do not have a proof at the moment, but one might try to show that for any $F$ of finite length with $S_1$ in its top, at least one of the three cases occurs: 1) all composition factors of $F$ are isomorphic to $S_1$; 2) the socle of $F$ contains a copy of $S_2$; or 3) the length-$2$ uniserial module with composition factors $S_1, S_2$ is a quotient of $F$ and the length-$2$ uniserial with composition factors $S_2, S_1$ is a submodule of $F$. In cases 2) and 3) there will always be a nonzero map from $F$ to $M \otimes_R F$. (Note: this suggestion is based on my belief that $R$ is isomorphic to the completed path algebra of the quiver with 2 vertices, a loop at each vertex and a pair of arrows connecting the vertices in each direction, modulo the relations that the two paths of length 2 from each vertex to the other are equal. If this is incorrect, then the structure of the finite-length $R$-modules may be more complicated than I envision.)
We can also see that for a cyclic module $F$, in order for the map $\mbox{Hom}_R(F,M\otimes_R F) \rightarrow \mbox{Hom}_R(R,M \otimes_R F)$ to be an isomorphism, there must be an epimorphism $F \rightarrow M \otimes_R F$, which would have to be an isomorphism since these two modules have the same length.