Let $M$ be an $(R, R)$-bimodule and let $S$ be the square-zero extension $R \oplus M$ of $R$ with multiplication $$(r_1 \oplus m_1)(r_2 \oplus m_2) = r_1 r_2 \oplus (m_1 r_2 + r_1 m_2).$$
If $M$ is free as a left $R$-module then so is $S$, and if $S$ is free as a right $R$-module then $M$ is projective. So it suffices to find $M$ which is free as a left $R$-module but not projective as a right $R$-module.
So let $R = T \times T$ where $T$ is some nonzero ring and let $M = R$ with $R$ acting on the left by left multiplication but acting on the right as follows: the second factor of $T$ acts diagonally by right multiplication and the first factor of $T$ acts trivially. Then $M$ is free as a left $R$-module but torsion as a right $R$-module.