Here is one example. Let $S$ denote the set of positive prime integers. Let $A$ be $\mathbb{Z}$. Let $R$ be the countably generated polynomial ring over $\mathbb{Z}$, $$R = \mathbb{Z}[\{x_p:p\in S\}].$$ Let $I\subset R$ be the ideal generated by $\{px_p : p\in S\}$. Let $B$ be $R/I$. Then the ideal $\mathfrak{p}$ of $B$ generated by $\{x_p : p\in S\}$ is prime, since $B/\mathfrak{p}$ is just the integral domain $\mathbb{Z}$. For every integer $p\in S$, $p$ is not in $\mathfrak{p}$. Hence, the localization of $B$ at $\mathfrak{p}$ factors through $B\otimes_{\mathbb{Z}}\mathbb{Q}$. But, of course, this localization is already $(B/\mathfrak{p})\otimes_{\mathbb{Z}}\mathbb{Q}$, which is $\mathbb{Q}$. Since $\mathbb{Q}$ is flat over $\mathbb{Z}$, thus $\mathfrak{p}$ is in $U$.

Every Zariski open subset of $\text{Spec}(B)$ containing $\mathfrak{p}$ contains a basic open subset of the form, $$D(b)=\{\mathfrak{q} \in \text{Spec}(B) : b\not\in \mathfrak{q}\},$$ for some $b\in B\setminus \mathfrak{p}$. Since $b$ is not in $\mathfrak{p}$, $b$ equals $n+c$ for some $c\in \mathfrak{p}$ and for some nonzero $n\in \mathbb{Z}$. Of course $c$ is a polynomial in only finitely many of the variables $x_p$. Also $n$ is divisible by only finitely many primes. Hence there exists a prime $q$ such that $c$ does not involve $x_q$, and, also, $q$ does not divide $n$.

Consider the unique surjective $\mathbb{Z}$-algebra homomorphism, $$ u_q : R \to (\mathbb{Z}/q\mathbb{Z})[x_q],\ \ u_q(x_q) = x_q, \ \ u_q(x_p) = 0,\ p\neq q. $$ Clearly $\text{Ker}(u_q)$ contains $px_p$ for every $p\neq q$, since $u_q(x_p)$ equals $0$. But since $q$ equals $0$ in $\mathbb{Z}/q\mathbb{Z}$, also $\text{Ker}(u_q)$ contains $qx_q$. Hence $u_q$ factors through a unique surjective ring homomorphism, $$ \overline{u}_q: B \to (\mathbb{Z}/q\mathbb{Z})[x_q].$$ Since $(\mathbb{Z}/q\mathbb{Z})[x_q]$ is an integral domain, the ideal $\mathfrak{q}:=\text{Ker}(\overline{u}_q)$ is a prime ideal of $B$. Also, since $\overline{u}_q(c)$ equals $0$, $\overline{u}_q(b)$ equals $\overline{u}_q(n)$. Since $q$ does not divide $n$, $\overline{u}_q(b)$ is nonzero. Thus $\mathfrak{q}$ is in $D(b)$.

Of course for every prime $p\neq q$, since $\overline{u}_q(p)$ is nonzero, also $p$ is not in $\mathfrak{q}$. Thus the localization $B\to B_{\mathfrak{q}}$ factors through $B\otimes_{\mathbb{Z}} \mathbb{Z}_{(q)}$, which is clearly just $\mathbb{Z}_{(q)}[x_q]/\langle qx_q \rangle$. Of course the image of $\mathfrak{q}$ in this localization is the principal ideal $\langle q \rangle$. In particular, $x_q$ is not in this prime ideal. Hence, the localization inverts $x_q$, and thus annihilates $q$. So $B_{\mathfrak{q}}$ is simply the field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ of rational functions in the variable $x_q$ over the field $\mathbb{Z}/q\mathbb{Z}$.

The field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ is not flat over $\mathbb{Z}$, since $q$ is a zerodivisor. Thus $\mathfrak{q}$ is not in $U$. Therefore $D(b)$ is not contained in $U$. Since this holds for every $b\in B\setminus \mathfrak{p}$, $U$ contains no Zariski open neighborhood of $\mathfrak{p}$, even though $U$ contains $\mathfrak{p}$. Therefore $U$ is not a Zariski open subset of $\text{Spec}(B)$.

Edit. In fact, it is not hard to see that $U$ is precisely $\{\mathfrak{p}\}$ for this ring.

Here is one example. Let $S$ denote the set of positive prime integers. Let $A$ be $\mathbb{Z}$. Let $R$ be the countably generated polynomial ring over $\mathbb{Z}$, $$R = \mathbb{Z}[\{x_p:p\in S\}].$$ Let $I\subset R$ be the ideal generated by $\{px_p : p\in S\}$. Let $B$ be $R/I$. Then the ideal $\mathfrak{p}$ of $B$ generated by $\{x_p : p\in S\}$ is prime, since $B/\mathfrak{p}$ is just the integral domain $\mathbb{Z}$. For every integer $p\in S$, $p$ is not in $\mathfrak{p}$. Hence, the localization of $B$ at $\mathfrak{p}$ factors through $B\otimes_{\mathbb{Z}}\mathbb{Q}$. But, of course, this localization is already $(B/\mathfrak{p})\otimes_{\mathbb{Z}}\mathbb{Q}$, which is $\mathbb{Q}$. Since $\mathbb{Q}$ is flat over $\mathbb{Z}$, thus $\mathfrak{p}$ is in $U$.

Every Zariski open subset of $\text{Spec}(B)$ containing $\mathfrak{p}$ contains a basic open subset of the form, $$D(b)=\{\mathfrak{q} \in \text{Spec}(B) : b\not\in \mathfrak{q}\},$$ for some $b\in B\setminus \mathfrak{p}$. Since $b$ is not in $\mathfrak{p}$, $b$ equals $n+c$ for some $c\in \mathfrak{p}$ and for some nonzero $n\in \mathbb{Z}$. Of course $c$ is a polynomial in only finitely many of the variables $x_p$. Also $n$ is divisible by only finitely many primes. Hence there exists a prime $q$ such that $c$ does not involve $x_q$, and, also, $q$ does not divide $n$.

Consider the unique surjective $\mathbb{Z}$-algebra homomorphism, $$ u_q : R \to (\mathbb{Z}/q\mathbb{Z})[x_q],\ \ u_q(x_q) = x_q, \ \ u_q(x_p) = 0,\ p\neq q. $$ Clearly $\text{Ker}(u_q)$ contains $px_p$ for every $p\neq q$, since $u_q(x_p)$ equals $0$. But since $q$ equals $0$ in $\mathbb{Z}/q\mathbb{Z}$, also $\text{Ker}(u_q)$ contains $qx_q$. Hence $u_q$ factors through a unique surjective ring homomorphism, $$ \overline{u}_q: B \to (\mathbb{Z}/q\mathbb{Z})[x_q].$$ Since $(\mathbb{Z}/q\mathbb{Z})[x_q]$ is an integral domain, the ideal $\mathfrak{q}:=\text{Ker}(\overline{u}_q)$ is a prime ideal of $B$. Also, since $\overline{u}_q(c)$ equals $0$, $\overline{u}_q(b)$ equals $\overline{u}_q(n)$. Since $q$ does not divide $n$, $\overline{u}_q(b)$ is nonzero. Thus $\mathfrak{q}$ is in $D(b)$.

Of course for every prime $p\neq q$, since $\overline{u}_q(p)$ is nonzero, also $p$ is not in $\mathfrak{q}$. Thus the localization $B\to B_{\mathfrak{q}}$ factors through $B\otimes_{\mathbb{Z}} \mathbb{Z}_{(q)}$, which is clearly just $\mathbb{Z}_{(q)}[x_q]/\langle qx_q \rangle$. Of course the image of $\mathfrak{q}$ in this localization is the principal ideal $\langle q \rangle$. In particular, $x_q$ is not in this prime ideal. Hence, the localization inverts $x_q$, and thus annihilates $q$. So $B_{\mathfrak{q}}$ is simply the field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ of rational functions in the variable $x_q$ over the field $\mathbb{Z}/q\mathbb{Z}$.

The field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ is not flat over $\mathbb{Z}$, since $q$ is a zerodivisor. Thus $\mathfrak{q}$ is not in $U$. Therefore $D(b)$ is not contained in $U$. Since this holds for every $b\in B\setminus \mathfrak{p}$, $U$ contains no Zariski open neighborhood of $\mathfrak{p}$, even though $U$ contains $\mathfrak{p}$. Therefore $U$ is not a Zariski open subset of $\text{Spec}(B)$.

Edit. In fact, it is not hard to see that $U$ is precisely $\{\mathfrak{p}\}$ for this ring.

Second Edit. I realize now that the ring $B$ above is "almost" a subring of the ring proposed by user76758 in the comments (I did not see that proposal until after I posted). Let $J$ be the ideal in $R$ generated by $px_p$ and $x_p^2-x_p$ for every $p$ in $S$. Then $C=R/J$ is still a counterexample, for essentially the same reason as above. Also $C$ is isomorphic to the $\mathbb{Z}$-subalgebra of $\prod_p \mathbb{F}_p$ generated by every element $\overline{x_p}$ that has coordinate $1$ in the $p$-factor and that has $0$ in every other factor.

Here is one example. Let $S$ denote the set of positive prime integers. Let $A$ be $\mathbb{Z}$. Let $R$ be the countably generated polynomial ring over $\mathbb{Z}$, $$R = \mathbb{Z}[\{x_p:p\in S\}].$$ Let $I\subset R$ be the ideal generated by $\{px_p : p\in S\}$. Let $B$ be $R/I$. Then the ideal $\mathfrak{p}$ of $B$ generated by $\{x_p : p\in S\}$ is prime, since $B/\mathfrak{p}$ is just the integral domain $\mathbb{Z}$. For every integer $p\in S$, $p$ is not in $\mathfrak{p}$. Hence, the localization of $B$ at $\mathfrak{p}$ factors through $B\otimes_{\mathbb{Z}}\mathbb{Q}$. But, of course, this localization is already $(B/\mathfrak{p})\otimes_{\mathbb{Z}}\mathbb{Q}$, which is $\mathbb{Q}$. Since $\mathbb{Q}$ is flat over $\mathbb{Z}$, thus $\mathfrak{p}$ is in $U$.

Every Zariski open subset of $\text{Spec}(B)$ containing $\mathfrak{p}$ contains a basic open subset of the form, $$D(b)=\{\mathfrak{q} \in \text{Spec}(B) : b\not\in \mathfrak{q}\},$$ for some $b\in B\setminus \mathfrak{p}$. Since $b$ is not in $\mathfrak{p}$, $b$ equals $n+c$ for some $c\in \mathfrak{p}$ and for some nonzero $n\in \mathbb{Z}$. Of course $c$ is a polynomial in only finitely many of the variables $x_p$. Also $n$ is divisible by only finitely many primes. Hence there exists a prime $q$ such that $c$ does not involve $x_q$, and, also, $q$ does not divide $n$.

Consider the unique surjective $\mathbb{Z}$-algebra homomorphism, $$ u_q : R \to (\mathbb{Z}/q\mathbb{Z})[x_q],\ \ u_q(x_q) = x_q, \ \ u_q(x_p) = 0,\ p\neq q. $$ Clearly $\text{Ker}(u_q)$ contains $px_p$ for every $p\neq q$, since $u_q(x_p)$ equals $0$. But since $q$ equals $0$ in $\mathbb{Z}/q\mathbb{Z}$, also $\text{Ker}(u_q)$ contains $qx_q$. Hence $u_q$ factors through a unique surjective ring homomorphism, $$ \overline{u}_q: B \to (\mathbb{Z}/q\mathbb{Z})[x_q].$$ Since $(\mathbb{Z}/q\mathbb{Z})[x_q]$ is an integral domain, the ideal $\mathfrak{q}:=\text{Ker}(\overline{u}_q)$ is a prime ideal of $B$. Also, since $\overline{u}_q(c)$ equals $0$, $\overline{u}_q(b)$ equals $\overline{u}_q(n)$. Since $q$ does not divide $n$, $\overline{u}_q(b)$ is nonzero. Thus $\mathfrak{q}$ is in $D(b)$.

Of course for every prime $p\neq q$, since $\overline{u}_q(p)$ is nonzero, also $p$ is not in $\mathfrak{q}$. Thus the localization $B\to B_{\mathfrak{q}}$ factors through $B\otimes_{\mathbb{Z}} \mathbb{Z}_{(q)}$, which is clearly just $\mathbb{Z}_{(q)}[x_q]/\langle qx_q \rangle$. Of course the image of $\mathfrak{q}$ in this localization is the principal ideal $\langle q \rangle$. In particular, $x_q$ is not in this prime ideal. Hence, the localization inverts $x_q$, and thus annihilates $q$. So $B_{\mathfrak{q}}$ is simply the field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ of rational functions in the variable $x_q$ over the field $\mathbb{Z}/q\mathbb{Z}$.

The field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ is not flat over $\mathbb{Z}$, since $q$ is a zerodivisor. Thus $\mathfrak{q}$ is not in $U$. Therefore $D(b)$ is not contained in $U$. Since this holds for every $b\in B\setminus \mathfrak{p}$, $U$ contains no Zariski open neighborhood of $\mathfrak{p}$, even though $U$ contains $\mathfrak{p}$. Therefore $U$ is not a Zariski open subset of $\text{Spec}(B)$.

Here is one example. Let $S$ denote the set of positive prime integers. Let $A$ be $\mathbb{Z}$. Let $R$ be the countably generated polynomial ring over $\mathbb{Z}$, $$R = \mathbb{Z}[\{x_p:p\in S\}].$$ Let $I\subset R$ be the ideal generated by $\{px_p : p\in S\}$. Let $B$ be $R/I$. Then the ideal $\mathfrak{p}$ of $B$ generated by $\{x_p : p\in S\}$ is prime, since $B/\mathfrak{p}$ is just the integral domain $\mathbb{Z}$. For every integer $p\in S$, $p$ is not in $\mathfrak{p}$. Hence, the localization of $B$ at $\mathfrak{p}$ factors through $B\otimes_{\mathbb{Z}}\mathbb{Q}$. But, of course, this localization is already $(B/\mathfrak{p})\otimes_{\mathbb{Z}}\mathbb{Q}$, which is $\mathbb{Q}$. Since $\mathbb{Q}$ is flat over $\mathbb{Z}$, thus $\mathfrak{p}$ is in $U$.

Every Zariski open subset of $\text{Spec}(B)$ containing $\mathfrak{p}$ contains a basic open subset of the form, $$D(b)=\{\mathfrak{q} \in \text{Spec}(B) : b\not\in \mathfrak{q}\},$$ for some $b\in B\setminus \mathfrak{p}$. Since $b$ is not in $\mathfrak{p}$, $b$ equals $n+c$ for some $c\in \mathfrak{p}$ and for some nonzero $n\in \mathbb{Z}$. Of course $c$ is a polynomial in only finitely many of the variables $x_p$. Also $n$ is divisible by only finitely many primes. Hence there exists a prime $q$ such that $c$ does not involve $x_q$, and, also, $q$ does not divide $n$.

Consider the unique surjective $\mathbb{Z}$-algebra homomorphism, $$ u_q : R \to (\mathbb{Z}/q\mathbb{Z})[x_q],\ \ u_q(x_q) = x_q, \ \ u_q(x_p) = 0,\ p\neq q. $$ Clearly $\text{Ker}(u_q)$ contains $px_p$ for every $p\neq q$, since $u_q(x_p)$ equals $0$. But since $q$ equals $0$ in $\mathbb{Z}/q\mathbb{Z}$, also $\text{Ker}(u_q)$ contains $qx_q$. Hence $u_q$ factors through a unique surjective ring homomorphism, $$ \overline{u}_q: B \to (\mathbb{Z}/q\mathbb{Z})[x_q].$$ Since $(\mathbb{Z}/q\mathbb{Z})[x_q]$ is an integral domain, the ideal $\mathfrak{q}:=\text{Ker}(\overline{u}_q)$ is a prime ideal of $B$. Also, since $\overline{u}_q(c)$ equals $0$, $\overline{u}_q(b)$ equals $\overline{u}_q(n)$. Since $q$ does not divide $n$, $\overline{u}_q(b)$ is nonzero. Thus $\mathfrak{q}$ is in $D(b)$.

Of course for every prime $p\neq q$, since $\overline{u}_q(p)$ is nonzero, also $p$ is not in $\mathfrak{q}$. Thus the localization $B\to B_{\mathfrak{q}}$ factors through $B\otimes_{\mathbb{Z}} \mathbb{Z}_{(q)}$, which is clearly just $\mathbb{Z}_{(q)}[x_q]/\langle qx_q \rangle$. Of course the image of $\mathfrak{q}$ in this localization is the principal ideal $\langle q \rangle$. In particular, $x_q$ is not in this prime ideal. Hence, the localization inverts $x_q$, and thus annihilates $q$. So $B_{\mathfrak{q}}$ is simply the field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ of rational functions in the variable $x_q$ over the field $\mathbb{Z}/q\mathbb{Z}$.

The field $(\mathbb{Z}/q\mathbb{Z})(x_q)$ is not flat over $\mathbb{Z}$, since $q$ is a zerodivisor. Thus $\mathfrak{q}$ is not in $U$. Therefore $D(b)$ is not contained in $U$. Since this holds for every $b\in B\setminus \mathfrak{p}$, $U$ contains no Zariski open neighborhood of $\mathfrak{p}$, even though $U$ contains $\mathfrak{p}$. Therefore $U$ is not a Zariski open subset of $\text{Spec}(B)$.

Edit. In fact, it is not hard to see that $U$ is precisely $\{\mathfrak{p}\}$ for this ring.