The comment of @MatthieuRomagny gives a link to some positive answers under additional hypotheses. Nonetheless, the answer is negative without further hypotheses. Before stating the negative counterexamples, let me state the positive result (that I suspect motivated this question).
Zariski's Main Theorem (Original Form). A birational (separated, finite type) morphism to a normal (finite type) $k$-scheme restricts as an open immersion on the maximal open of the domain where the morphism is quasi-finite.
One excellent reference for Zariski's Main Theorem is Section III.9 of the following (the formulation above is on p. 288).
MR0971985 (89k:14001)
Mumford, David
The red book of varieties and schemes.
Lecture Notes in Mathematics, 1358.
Springer-Verlag, Berlin, 1988.
Corollary of ZMT. For a birational (separated, finite type) morphism to a normal (finite type) $k$-scheme, for every point of the target where the fiber is nonempty and finite, there exists an open neighborhood of the point over which the morphism is an isomorphism.
Counterexamples. Nonetheless, if you remove the hypothesis that the morphism is birational, there are counterexamples. Here is one.
Let $k$ be a field, and let $A$ be the polynomial ring $k[s,t,u]$. Let $B$ be the ring $k[x,y,z,w]/\langle zw-1 \rangle$. Both of these are finitely generated $k$-algebras that are regular, in fact $k$-smooth. Now consider the $k$-algebra homomorphism, $$p:A\to B, \ \ p(s) = x(1-x), \ p(t) = xy, \ p(u) = z(1-xz).$$ If we invert the element $s$ in $A$ and the image element $x(1-x)$ in $B$, then the induced ring homomorphism is finite and flat of rank $4$. In particular, the localized ring homomorphism is injective. Since the natural map from $A$ to $A[1/s]$ is injective, also $p$ is injective.
Finally, consider the prime ideal $P=\langle s,t,u\rangle$ in $A$. There is precisely one prime lying above this prime, namely $\langle x-1,y,z-1,w-1\rangle$. However, the prime ideal $Q=\langle s,t \rangle$ has infinitely many primes lying over it, e.g., $\langle x,q(y) \rangle$ where $q(y)\in k[y]$ is an arbitrary nonzero, noninvertible element that is irreducible. Since $Q$ is contained in $P$, for every $f\in A\setminus P$, also $f$ is in $A\setminus Q$. Thus, the ideal $QA_f$ is a prime ideal of $A_f$. So the induced ring homomorphism $A_f \to B_{p(f)}$ still admits a prime ideal $QA_f$ that has infinitely many primes lying over it in $B_{p(f)}$.
There are also counterexamples to the corollary if we allow that the fiber is empty (but the morphism is still birational). Namely, consider the self-map, $$r:A\to A, \ \ r(s) = s, \ r(t) = st, \ r(u) = 1-su.$$ This ring homomorphism is birational. The fiber over $\langle s,t,u\rangle$ is empty. Yet the fiber over $\langle s,t \rangle$ is infinite, as above.