Edit. I started this answer earlier, but then I had to do something else.
That conjecture is false without further hypotheses, e.g., flatness of $p$. Consider the case where $A$ equals $k[s]$, where $X$ is the effective Cartier divisor of $\mathbb{P}^3_A = \text{Proj}\ A[T,U,V,W]$ with defining equation $s(T^3+U^3+V^3+W^3)$, where $\mathcal{E}$ equals the structure sheaf of the closed subscheme $C$ of $X$ with defining ideal $\langle T+U,V+W \rangle$, and where $\mathcal{F}$ equals the pushforward to $X$ of the dualizing sheaf of $C$. For $i=0$, the formation of $\text{Hom}_{\mathcal{O}_X}(\mathcal{E},\mathcal{F})$ is compatible with arbitrary base change: it is always zero.
Now consider the functor $\text{Ext}^1_{p_B}(\mathcal{E},\mathcal{F})$. For the base change to $A[1/s]$, it is representable by a locally free sheaf $Q_{1/s}$ of rank $1$. Also, for the base change to $A/sA$, it is representable by a locally free sheaf of rank $1$. So if your conjecture is correct, then the functor is representable by a locally free sheaf $Q$ of rank $1$, i.e., the structure sheaf.
But then the same hypotheses apply for the functor $\text{Ext}^2_{p_B}(\mathcal{E},\mathcal{F})$. However, now the base change to $A[1/s]$ is representable by a locally free sheaf of rank $2$, whereas the base change to $A/sA$ is representable by a locally free sheaf of rank $1$. There is no finitely generated $A$-module $R$ such that $R\otimes_A A[1/s]$ is free of rank $2$ yet $R/sR$ is free of rank $1$.