Answering my own question in comments:
Let $k$ be a field, and $R=k\oplus V$$A=k\oplus V$, where $V$ is an infinite-dimensional square-zero ideal. Then I think $R$$A$ has no non-projective $FP_3$-modules (using your terminology), as the kernel of any map between finitely generated free modules, that's not a surjection onto a direct summand, has infinitely generated kernel.
So every module$A$-module $M$ satisfies your condition, but there are non-flat modules.