In other words, if $I$ is the kernel of $B\to R$, must $IA\cap R=0$?
I know that the kernel of $R\to A\otimes_B R = A/IA$ must consist of nilpotents by the following argument: since $A$ is finite as an $R$-module it is integral as an $R$-modulealgebra, and hence also as a $B$-modulealebra. Since integral extensions have the Lying Over property, the map of schemes $\mathrm{Spec}(A)\to \mathrm{Spec}(B)$ is surjective. Surjectivity is preserved by base extension, so the map of schemes $\mathrm{Spec}(A\otimes_B R) \to \mathrm{Spec}(R)$ is surjective. And a morphism of affine schemes has dense image if and only if the kernel of the corresponding ring homomorphism consists of nilpotents.
Here's a counterexample (due to Hendrik Lenstra): let $A=\mathbb{Z}[x]/(x^2-x)$ and $B=\mathbb{Z}[y]/(y^2-2y)$; the homomorphism $B\to A: y\mapsto 2x$ is injective. However, tensoring with the $B$-algebra $B/(2)$, we obtain a non-injective map $B/(2)\to A/(2): y\mapsto 0$. The original question asks for an example in which the map $B\to R$ is a section of an $R$-algebra structure on $B$ making $B$ and $A$ free of finite rank as $R$-modules.
Update: For a positive result (also from conversations with Hendrik Lenstra), the statement holds if $B$ is generate by a single element as an $R$-module, which (since $B$ is also free of finite rank as an $R$-module) implies that $B$ is of the form $R[x]/(f(x))$ with $f$ monic.
The argument goes like this: Say $f$ has degree $n$ and the homomorphism $B\cong R[x]/(f(x))\to R$ sends $x$ to $r$; then by choosing the generator $x-r$ of $B$ instead of $x$, we may assume $x\mapsto 0$. In that case, the kernel $I$ of $B\to R$ is $(x)$, so we find that $f(x)\in R[x]$ is divisible by $x$; write it as $f(x) = x\cdot g(x)$ for some monic polynomial $g$ of degree $n-1$. Now suppose we have something in $IA\cap R$: it will be an equation $xa = s$ for some $a\in A$ and $s\in R$. Multiplying both sides by $g(x)$, we obtain $s\cdot g(x) = xg(x) a = f(x) a = 0$ in $A$, since $f(x)=0$ in $B$. But $s\cdot g(x)=0$ is an $R$-linear relation among $\{1,x,\dots,x^{n-1}\}$, which form an $R$-basis of $B$, so all the coefficients must be zero. But $g$ is monic, so the coefficient of $x^{n-1}$ in $s\cdot g(x)$ is $s$, so $s=0$. Thus $IA\cap R=0$, so $R\to A/IA \cong A\otimes_B R$ is injective.
However, finite algebras are generally not monogenic, so there may still be counterexamples.