If $f:X\to Y$ is separated of finite type between noetherian schemes and $f_*$ preserves coherence, then $f$ is proper. Here is a proof that follows the geometric idea (given in the comments of Piotr Achinger and Karl Schwede) to use a curve in $X$ that is not proper over $Y$ :

  Since it is possible to extend a coherent sheaf on an open of $X$ to the whole of $X$ [Stacks project, Tag 01PD], we see that the statement is local on $Y$, and we are reduced to the case where $Y$ is the spectrum of a local ring.

  By Nagata's compactification theorem, we may find a compactification of $f$: it is the composition of an open immersion with dense image $i:X\to \overline{X}$ and a proper morphism $\overline{f}:\overline{X}\to Y$. We suppose for contradiction that $Z=\overline{X}\setminus X$ is not empty.

  Let us reduce to the case where $Z$ is finite and for every $z\in Z$, $\mathcal{O}_{\overline{X},z}$ is of dimension $1$. We do it by applying iteratively the following procedure. Choose a closed point $z\in Z$ at which $\mathcal{O}_{\overline{X},z}$ is not of dimension $1$. Using prime avoidance, choose a function $f\in \mathcal{O}_{\overline{X},z}$ that does not vanish on any irreducible component of $\overline{X}$ through $z$ nor on any positive-dimensional irreducible component of $Z$ through $z$. Replace $\overline{X}$, $X$ and $Z$ by their intersection with the closure of $\{f=0\}$ in $\overline{X}$. The hypothesis of preserving coherence still holds because closed immersions preserve coherence. The Hauptidealsatz ensures that $X$ is still dense in $\overline{X}$.

Using noetherian induction on $Z$, and induction on the dimensions of the local rings $(\mathcal{O}_{\overline{X},z})_{z\in Z}$, we see that the procedure terminates. Hence, from now on, we suppose that $Z$ is finite and for every $z\in Z$, $\mathcal{O}_{\overline{X},z}$ is of dimension $1$. Replacing moreover $\overline{X}$ by one of its components through $z$, we may assume that it is integral of generic point $\eta$.

Now we distinguish two cases. If $f(\eta)$ is the closed point of $Y$, then $X$ is an affine curve over the residue field of $Y$, and $f_*\mathcal{O}_X$ is not coherent: a contradiction.

Otherwise, consider the Stein factorization $\overline{X}\to S\to Y$ of $\overline{f}$. Up to replacing $Y$ by $S$ (we do not lose the hypothesis of preserving coherence, because $S\to Y$ is finite), we may suppose that $\overline{f}$ satisfies $\overline{f}_*\mathcal{O}=\mathcal{O}$. In particular, it has connected fibers.

The points of $Z$ are closed points, hence sent to closed points of $Y$ by properness of $\overline{f}$. Since their only generization in $\overline{X}$ is $\eta$, and since the fibers of $\overline{f}$ are connected, each of them is a fiber of $\overline{f}$. It follows that $X$ is the pull-back by $\overline{f}$ of an open subset $j:U\hookrightarrow Y$.

Choose $z\in Z$ and let $y:=\overline{f}(z)$. Then $\mathcal{O}_{Y,y}$ is of dimension $1$. Indeed, a generization of $y$ different from the generic point of $Y$ would have preimage $x$ in $\overline{X}$ because $\overline{f}$ is proper dominant, hence surjective. Using that $z=\overline{f}^{-1}(y)$, one sees that $z$ is a specialization of $x$, contradicting the fact that $\mathcal{O}_{\overline{X},z}$ is of dimension $1$.

Let $g:X\to U$ be the restriction of $\overline{f}$ to $X$. Since $g_*\mathcal{O}=\mathcal{O}$ and $f_*$ preserves coherence, $j_*\mathcal{O}$ is coherent.
Localizing at $y$, it implies that $\textrm{Frac}(\mathcal{O}_{Y,y})$ is of finite type over $\mathcal{O}_{Y,y}$, which is impossible.