The comment of @MatthieuRomagny gives a link to some positive answers under additional hypotheses. Nonetheless, the answer is negative without further hypotheses. Before stating the negative counterexamples, let me state the positive result (that I suspect motivated this question).

**Zariski's Main Theorem (Original Form).** A birational (separated, finite type) morphism to a normal (finite type) $k$-scheme restricts as an open immersion on the maximal open of the domain where the morphism is quasi-finite.

One excellent reference for Zariski's Main Theorem is Section III.9 of the following (the formulation above is on p. 288).

MR0971985 (89k:14001)

Mumford, David

The red book of varieties and schemes.

Lecture Notes in Mathematics, 1358.

Springer-Verlag, Berlin, 1988.

**Corollary of ZMT.** For a birational (separated, finite type) morphism to a normal (finite type) $k$-scheme, for every point of the target where the fiber is nonempty and finite, there exists an open neighborhood of the point over which the morphism is an isomorphism.

**Counterexamples.** Nonetheless, if you remove the hypothesis that the morphism is birational, there are counterexamples. Here is one.

Let $k$ be a field, and let $A$ be the polynomial ring $k[s,t,u]$. Let $B$ be the ring $k[x,y,z,w]/\langle zw-1 \rangle$. Both of these are finitely generated $k$-algebras that are regular, in fact $k$-smooth. Now consider the $k$-algebra homomorphism, $$p:A\to B, \ \ p(s) = x(1-x), \ p(t) = xy, \ p(u) = z(1-xz).$$ If we invert the element $s$ in $A$ and the image element $x(1-x)$ in $B$, then the induced ring homomorphism is finite and flat of rank $4$. In particular, the localized ring homomorphism is injective. Since the natural map from $A$ to $A[1/s]$ is injective, also $p$ is injective.

Finally, consider the prime ideal $P=\langle s,t,u\rangle$ in $A$. There is precisely one prime lying above this prime, namely $\langle x-1,y,z-1,w-1\rangle$. However, the prime ideal $Q=\langle s,t \rangle$ has infinitely many primes lying over it, e.g., $\langle x,q(y) \rangle$ where $q(y)\in k[y]$ is an arbitrary nonzero, noninvertible element that is irreducible. Since $Q$ is contained in $P$, for every $f\in A\setminus P$, also $f$ is in $A\setminus Q$. Thus, the ideal $QA_f$ is a prime ideal of $A_f$. So the induced ring homomorphism $A_f \to B_{p(f)}$ still admits a prime ideal $QA_f$ that has infinitely many primes lying over it in $B_{p(f)}$.

There are also counterexamples to the corollary if we allow that the fiber is empty (but the morphism is still birational). Namely, consider the self-map, $$r:A\to A, \ \ r(s) = s, \ r(t) = st, \ r(u) = 1-su.$$ This ring homomorphism is birational. The fiber over $\langle s,t,u\rangle$ is empty. Yet the fiber over $\langle s,t \rangle$ is infinite, as above.