Cohomological base change $\require{AMScd}$
Consider the Cartesian diagram  of Noetherian schemes and commutative rings $R$, $R'$:
\begin{CD}
N' @>{h'}>> N\\@VV{g'}V @VVgV\\ M' @>h>> M \\ @VVV @VVV \\ \mathbf{Spec R'} @>>{i}> \mathbf{Spec} R
\end{CD}
Suppose $g$ is proper. For a coherent sheaf $\mathcal{F}$ on $N$, if $R^ig_*\mathcal{F}$ is flat over $\mathbf{Spec} R$ for any $i\in \mathbb{N}$, then is there an isoorphism
$$
h^*R^ig_*\mathcal{F} \simeq R^i{g'}_*{h'}^*\mathcal{F}
$$
for any $i$?
 A: Basically, you are asking for a weakening of the hypothesis for semi-continuity to only flatness of the higher direct images of $\mathcal{F}$ rather than flatness of $\mathcal{F}$ itself. This is not possible as the following example shows. There may be even simpler examples!
Take $\pi:Y\to \mathbb{A}^2$ to be the blow-up at the origin with exceptional fibre $E$, and $\mathcal{G}$ to be the sheaf $\mathcal{O}_Y(E)$.
Take $M=\mathbf{Spec}(R)=\mathbb{A}^1$ to be the line $x=0$ in $\mathbb{A}^2$ passing through the origin. Take $N=\pi^{-1}(M)$, $g=\pi_{|_N}$ and $\mathcal{F}=\mathcal{G}\otimes\mathcal{O}_N=\mathcal{O}_N(E)$.
We use the sequence
$$
    0 \to \mathcal{O}_Y(-N+E)\to\mathcal{O}_Y(E)\to\mathcal{O}_N(E)\to 0
$$
to compute the higher direct images of $\mathcal{F}=\mathcal{O}_N(E)$. Note that $\mathcal{O}_Y(-N)$ is $x\mathcal{O}_Y\subset\mathcal{O}_Y$ since it is $\pi^{*}\mathcal{O}_{\mathbb{A}^2}(-M)$ and we have $\mathcal{O}_{\mathbb{A}^2}(-M)=x\mathcal{O}_{\mathbb{A}^2}$. Thus, we only need to compute
$$
    R^i\pi_*\mathcal{O}_Y(E)
    =
    \begin{cases}
        \mathcal{O}_{\mathbb{A}^2} & i=0 \\
        0 & ~\text{otherwise}
    \end{cases}
$$
It follows that $R^{i}g_*\mathcal{F}$ is $0$ for $i>0$ and $\mathcal{O}_{\mathbb{A}^1}$ for $i=0$. In particular, it is flat over $R$ for all $i$.
Now take $\mathbf{Spec}(R')$ to be the origin in $\mathbb{A}^1$. We see that $h'^{*}\mathcal{F}$ is the sheaf $\mathcal{O}_{\mathbb{P}^1}(-1)$ on $N'=\mathbb{P}^1$. It follows that $g'_{*}h'^{*}\mathcal{F}$
is the $0$ sheaf since $\mathcal{O}_{\mathbb{P}^1}(-1)$ has no global sections.
