Edit. I started this answer earlier, but then I had to do something else.
Positive answer.
Without any further hypothesis, the answer is positive if $i$ equals $0$. This is one of the theorems the follows from the "Exchange Property" in Section 7.7 of EGA III_2. If $i$ equals $1$, at least there is a well-defined functor: because $\mathcal{E}$ is $A$-flat, for every short exact sequence in the Yoneda-Ext group $\text{Ext}^1_{\mathcal{O}_{X}}(\mathcal{E},\mathcal{F})$, the base change to $B$ is still a short exact sequence.
When $X$ is $A$-flat, there are well-defined functors for every $i$. Moreover, there is a bounded below complex whose terms are finitely generated, locally free $A$-modules whose homology modules compute the Ext groups compatibly with arbitrary base change.
Since $p$ is projective, there exists a $p$-ample invertible sheaf $\mathcal{O}_X(1)$. Thus, there exists a bounded above resolution of $\mathcal{E}$ by $\mathcal{O}_X$-modules that are finite direct sums of twists $\mathcal{O}_X(-d)$ with $d$ so positive that $\mathcal{F}(d)$ has vanishing higher direct image sheaves for $p$. Thus, we can compute $R\textit{Hom}_{\mathcal{O}_X}(\mathcal{E},\mathcal{F})$ by forming the usual sheaf Hom of this resolution into $\mathcal{F}$. By construction, this new bounded below complex has terms that are finite direct sums of coherent sheaves $\mathcal{F}(d)$ having vanishing higher direct image sheaves. Thus, $Rp_*$ is just $p_*$. Altogether, this gives a bounded below complex of finite free $A$-modules whose homologies compute the Ext groups, compatible with arbitrary base change (since everything, including the sheaves $\mathcal{O}_X(-d)$, is $A$-flat).
Negative answer.
For $i\geq 2$, the conjecture is false without further hypotheses, e.g., flatness of $p$. If you assume that $p$ is flat, then I believe the conjecture is true (I will try to return to this soon).
Consider the case where $A$ equals $k[s]$, where $X$ is the effective Cartier divisor of $\mathbb{P}^3_A = \text{Proj}\ A[T,U,V,W]$ with defining equation $s(T^3+U^3+V^3+W^3)$, where $\mathcal{E}$ equals the structure sheaf of the closed subscheme $C$ of $X$ with defining ideal $\langle T+U,V+W \rangle$, and where $\mathcal{F}$ equals the pushforward to $X$ of the dualizing sheaf of $C$. For $i=0$, the formation of $\text{Hom}_{\mathcal{O}_X}(\mathcal{E},\mathcal{F}) = H^0(C,\omega_C)$ is compatible with arbitrary base change: it is always zero since $C$ is just $\mathbb{P}^1_A$ and $\omega_C$ is just $\mathcal{O}_{\mathbb{P}^1}(-2)$.
The flattening stratification of $X$ is the locally closed partition of $\text{Spec}(A)$ into two subsets: the basic open subset $D(s) = \text{Spec}(A[1/s])$ and the closed point $\text{Zero}(s) = \text{Spec}(A/sA)$. Over each of these locally closed subschemes, the restriction of $X$ is smooth: either a smooth cubic surface over $D(s)$ or all of $\mathbb{P}^2$ over $\text{Zero}(s)$. Moreover, the curve $C$ is smooth over $\text{Spec}(A)$; just a line. Thus, over each of these locally closed subschemes, the curve $C$ is a "regular embedding" in $X$ with locally free normal sheaf $N_{C/X}$ (the sheaf Hom into $\mathcal{O}_C$ of the ideal sheaf of $C$ in $X$). Over $D(s)$, the locally free normal sheaf $N_{C/X}$ is just $\mathcal{O}_{\mathbb{P}^1}(-1)$. Over $\text{Zero}(s)$, the locally free normal sheaf $N_{C/X}$ is just $\mathcal{O}_{\mathbb{P}^1}(+1)^{\oplus 2}$.
In fact, in this example $C$ is a transverse intersection of divisors: it is a divisor in $X$ over $D(s)$, and it is an intersection of two hyperplanes in $X$ over $\text{Zero}(s)$. This means that the object $R\textit{Hom}_{\mathcal{O}_X}(\mathcal{O}_C,\mathcal{F})$ is particularly simple when restricted over $D(s)$ and over $\text{Zero}(s)$: it is quasi-isomorphic to a complex that is simply the direct sum of its homology sheaves (i.e., the differentials are all zero), and these homology sheaves are just $\mathcal{H}^q = \bigwedge^q_{\mathcal{O}_C}(N_{C/X})\otimes_{\mathcal{O}_C} \mathcal{F}$. Thus, $Rp_*$ (over each stratum) is also just the direct sum of the complexes $Rp_*(\bigwedge^q_{\mathcal{O}_C}(N_{C/X})\otimes_{\mathcal{O}_C} \mathcal{F})[-q]$. That makes it particularly easy to compute the ranks of the Ext groups over each stratum.
More precisely, over $D(s)$, the complex $R\textit{Hom}_{\mathcal{O}_X}(\mathcal{E},\mathcal{F})$ is just $\mathcal{O}_{\mathbb{P}^1}(-2)[0]\oplus \mathcal{O}_{\mathbb{P}^1}(-3)[-1]$. Thus $Rp_*R\textit{Hom}_{\mathcal{O}_X}(\mathcal{E},\mathcal{F})$ is just $H^1(\mathbb{P}^1,\mathcal{O}_{\mathbb{P}^1}(-2))[-1]\oplus H^1(\mathbb{P}^1,\mathcal{O}_{\mathbb{P}^1}(-3))[-2]$.
Similarly, over $\text{Zero}(s)$, the complex $R\textit{Hom}_{\mathcal{O}_X}(\mathcal{E},\mathcal{F})$ is just $\mathcal{O}_{\mathbb{P}^1}(-2)[0]\oplus \mathcal{O}_{\mathbb{P}^1}(-1)^{\oplus 2}[-1]\oplus \mathcal{O}_{\mathbb{P}^1}(0)[-2]$. Thus $Rp_*R\textit{Hom}_{\mathcal{O}_X}(\mathcal{E},\mathcal{F})$ is just $H^1(\mathbb{P}^1,\mathcal{O}_{\mathbb{P}^1}(-2))[-1]\oplus H^0(\mathbb{P}^1,\mathcal{O}_{\mathbb{P}^1})[-2]$.
Now consider the functor $\text{Ext}^1_{p_B}(\mathcal{E},\mathcal{F})$. For the base change to $A[1/s]$, it is representable by a locally free sheaf $Q_{1/s}$ of rank $1$. Also, for the base change to $A/sA$, it is representable by a locally free sheaf of rank $1$. So if your conjecture is correct, then the functor is representable by a locally free sheaf $Q$ of rank $1$.
But then the same hypotheses apply for the functor $\text{Ext}^2_{p_B}(\mathcal{E},\mathcal{F})$. However, now the base change to $A[1/s]$ is representable by a locally free sheaf of rank $2$, whereas the base change to $A/sA$ is representable by a locally free sheaf of rank $1$. There is no finitely generated $A$-module $R$ such that $R\otimes_A A[1/s]$ is free of rank $2$ yet $R/sR$ is free of rank $1$.
