If $f:X\to Y$ is separated of finite type between noetherian schemes and $f_*$ preserves coherence, then $f$ is proper. Here is a proof that follows the geometric idea (given in the comments of Piotr Achinger and Karl Schwede) to use a curve in $X$ that is not proper over $Y$ :
Since it is possible to extend a coherent sheaf on an open of $X$ to the whole of $X$ [Stacks project, Tag 01PD], we see that the statement is local on $Y$, and we are reduced to the case where $Y$ is the spectrum of a local ring.
By Nagata's compactification theorem, we may find a compactification of $f$: it is the composition of an open immersion with dense image $i:X\to \overline{X}$ and a proper morphism $\overline{f}:\overline{X}\to Y$. We suppose for contradiction that $Z=\overline{X}\setminus X$ is not empty.
Let us reduce to the case where $Z$ is finite and for every $z\in Z$, $\mathcal{O}_{\overline{X},z}$ is of dimension $1$. We do it by applying iteratively the following procedure. Choose a closed point $z\in Z$ at which $\mathcal{O}_{\overline{X},z}$ is not of dimension $1$. Using prime avoidance, choose a function $f\in \mathcal{O}_{\overline{X},z}$ that does not vanish on any irreducible component of $\overline{X}$ through $z$ nor on any positive-dimensional irreducible component of $Z$ through $z$. Replace $\overline{X}$, $X$ and $Z$ by their intersection with the closure of $\{f=0\}$ in $\overline{X}$. The hypothesis of preserving coherence still holds because closed immersions preserve coherence. The Hauptidealsatz ensures that $X$ is still dense in $\overline{X}$.
Using noetherian induction on $Z$, and induction on the dimensions of the local rings $(\mathcal{O}_{\overline{X},z})_{z\in Z}$, we see that the procedure terminates. Hence, from now on, we suppose that $Z$ is finite and for every $z\in Z$, $\mathcal{O}_{\overline{X},z}$ is of dimension $1$. Replacing moreover $\overline{X}$ by one of its components through $z$, we may assume that it is integral of generic point $\eta$.
Now we distinguish two cases. If $f(\eta)$ is the closed point of $Y$, then $X$ is an affine curve over the residue field of $Y$, and $f_*\mathcal{O}_X$ is not coherent: a contradiction.
Otherwise, consider the Stein factorization $\overline{X}\to S\to Y$ of $\overline{f}$. Up to replacing $Y$ by $S$ (we do not lose the hypothesis of preserving coherence, because $S\to Y$ is finite), we may suppose that $\overline{f}$ satisfies $\overline{f}_*\mathcal{O}=\mathcal{O}$. In particular, it has connected fibers. Localizing $Y$ at a point in the image of $Z$ as in the first step, we may still assume that $Y$ is local.
The points of $Z$ are closed points, hence sent to the closed point $y$ of $Y$ by properness of $\overline{f}$. Since their only generization in $\overline{X}$ is $\eta$, and since the fibers of $\overline{f}$ are connected, $\overline{f}^{-1}(y)$ consists of a single point $z=Z$. It then follows that $\overline{X}$ has only two points $z$ and $\eta$, and that $Y$ has also only two points: their images. Consequently, $\overline{f}$ is quasi-finite and proper, hence finite, and in fact an isomorphism because $\overline{f}_*\mathcal{O}=\mathcal{O}$.
At this point, the hypothesis that $f_*\mathcal{O}$ is coherent means exactly that $\textrm{Frac}(\mathcal{O}_{Y,y})$ is of finite type over $\mathcal{O}_{Y,y}$, which is absurd.