You are correct that this is nontrivial.
The natural map $B \to A_p$ defines a map $B_p \to A_p$. The composition $A_p \to B_p \to A_p$ is clearly the identity. This makes the surjectivity of $A_p \to B_p$ equivalent to the injectivity of $B_p \to A_p$, and also equivalent to both of those maps being isomorphisms.
The injectivity of $B_p \to A_p$ is the same as saying that any element in $B$ which is in the kernel of the natural map $B \to A_p$, that, is, vanishes on some affine open neighborhood of $A_p$, say $D(s)$, is a zero divisor with some element of $B$ which is not in $p$. It's not obvious to me that this element must be a power of $s$ - perhaps it could be some other element that does not vanish.
The naive proof of this only works if $U$ is quasicompact. It's also easy to extend this proof to the reduced case. Presumably the proof you found was roughly similar to this argument.
You can come up with slightly stronger conditions that include both the reduced and quasicompact cases that force this argument to work, basically a bound on how nilpotent things can get. For instance you can mandate that $U$ has a covering by affines of which all but finitely many are reduced. However none of these conditions are very satisfying.
It's easy to come up with an example of $f \in B$ whose stalk is trivial but is not a zero-divisor with a power of $s$. Just take $A=k[x_1,x_2,...,y_1,y_2,...]/((x_ix_j)^{i+j}(y_i-y_j))$. Let $U$ be the complement of the vanishing set of the ideal generated by all the $x_i$. Let $f=y_i-y_1$ whenever $x_i \neq 0$, then $f$ is well-defined by the sheaf condition, $f$ is zero on $D(x_1)$, but $f$ is not a zero divisor with any power of $x_1$. But it's not clear to me whether there is another function that $f$ is a zero divisor with.