The morphism $\mathbb{P}^2\setminus 0\to *$ does not preserve coherence as there exist non-proper curves. If $f_*$ preserves coherence after arbitrary noetherian base change, then $f$ is universally closed: the valuative criterion is easily seen to hold (as indicated by Piotr Achinger) since the fraction field of a DVR is not coherent. Actually, using Riemann–Zariski tricksHowever, it seemsturns out that we have:it is not necessary to involve base change and it is also possible to characterize properness in terms of $R^1f_*$ (cf comment by Tom Graber).
Theorem 2. Let $f\colon X\to Y$ be a morphism of finite type between noetherian schemes. Then
(i) $f$ is universally closed if and only if $f_*$ preserves coherence, and
(ii) $f$ is proper if and only if $f$ is universally closed$f_*$ and $R^1f_*$ preserves coherence.
I was a bit hasty and the proof is currently incorrect at several places. It can be fixed and I will do so soon.
Proof. (sketch) One direction The necessity of the conditions is Theorem 1. For the other direction, assume that $f$ is not universally closed. Then there exists a DVR $D$ and a morphism in $g\colon \operatorname{Spec} D\to Y$(i) and a closed immersionwell-known in $\operatorname{Spec} K(D) \to X\times_Y \operatorname{Spec} D$(ii). We may also assumeTo see that $K(D)$ is the residue field of the image ofthey are sufficient we need some lemmas that say that the generic pointvaluative criteria can be checked using curves in $Y$$X$. Replacing $Y$ with the closure of the image of $g$ we may assume thatA $Y$curve is integral witha $K(D)=K(Y)$$1$-dimensional noetherian integral scheme.
Note First we show that a dominant affine open immersion does not preserve coherence unless it is an isomorphism (if it preserves coherence it isenough with curves of finite hence a closed and open immersion, hence an isomorphism)type over $Y$.
Now, by the theory of Riemann–Zariski spaces, the DVRLemma 1. Let $D$ is$f\colon X\to Y$ be a local ring of the inverse limit of all blow-ups on $Y$ (when $Y$ ismorphism of finite type over a field, andbetween noetherian schemes. If $D$ is "algebraic" then a finite number of blow-ups$f$ is sufficient, cf. Kollar–Mori Lem. 2not universally closed (resp.45 not separated). By approximation, this means that we can find
then there exists
(i) a blow-up $Y'\to Y$;
(ii) an affine open immersion $U'\to Y'$;
(iii) an affine open immersionseparated curve C together with a morphism $V'\to U'$;$C\to Y$ of finite type; and
(ivii) a closed immersion $V'\to X\times_Y U'$.
The morphism $\operatorname{Spec} D\to Y$ factors through $U'$ andsubscheme $\operatorname{Spec} K(D)\to \operatorname{Spec} D$ is the pull-back$C'$ of $V'\to U'$ etc.
Now consider the closure $\overline{V'}$ in $X\times_Y Y'$. Then $V'=\overline{V'}\times_{Y'} U'$. This means$X\times_Y C$ such that the push-forward of the structure sheaf of $\overline{V'}$ to $Y'$ is not coherent (since the restriction to $U'$$C'$ is not coherent).
Finally, we pick a relatively ample line bundle $\mathcal{O}_{Y'}(1)$ for $Y'\to Y$. If we let $F=\mathcal{O}_{\overline{V'}}(n)\in \mathrm{Coh}(X\times_Y Y')$curve and pick a large $n$, then the push-forward to $Y'$$C'\to C$ is not coherentbirational and generated by global sections relative tonot surjective $Y$(resp. This means that the push-forward to $Y$ is not coherent. QEDseparated).
Remark 1. In basic non-separated examples $R^if_*$ do not preserve coherence and perhaps coherence of higher direct push-forwards does imply separatedness as Tom Graber says. For schemes and algebraic spaces, Raynaud–Gruson's Chow lemma essentially reduces this question to morphisms ofProof. By the form $f\colon X\to \mathbb{P}^n_Y \to Y$ where $f\colon X\to \mathbb{P}^n_Y$ is étale. Twistingvaluative criterion, this perhaps reduces the question to étale morphisms. This approach could perhaps also givethere exists a simpler proof of Theorem 2.
Remark 2. The situation for algebraic stacks is more subtle. Theorem 1 holds as mentioned above. Theorem 2 can be extended to stacks as well: we can at least letDVR $Y$ be Deligne–Mumford$D$, a morphism (or have quasi-finite and separated diagonal)$Y_0:=\operatorname{Spec} D\to Y$ and $X$ be arbitrary. One immediately reduces toan integral closed subscheme $Y$ a scheme by taking a finite flat covering. The same proof then works with the only difference$Z_0\subseteq X_0:=X\times_Y Y_0$ such that $V'\to U'$ becomes some morphism followed by$Z_0\to Y_0$ is an affine open immersion (hence: the push-forwardinclusion of the structure sheaf isgeneric point (resp. not coherentseparated). This is enough forIn the proof to work. Remark 1 is more subtle though. If Remark 1 holds for schemes/algebraic spaces and $X$ has a good moduli space thenlatter case, we can assume that there are two different sections of $X$ has coherent cohomology if and$Z_0\to Y_0$ that only ifagree over the generic point $X_\mathrm{gms}$ is proper, e.g., BGL_n has coherent cohomology but is not separated$U_0=\operatorname{Spec} K(D)$. This is all well-known
We now approximate the map $Y_0\to Y$ and one may argue that such stacks are "almost proper"the data (cf. paper by Halpern-Leistner–Preygel$U_0\subseteq Y_0$, arXiv$Z_0\to Y_0$:1402.3204). there exists
Moreover, we can arrange so that $Z'\to Y'$ is equal to $U'\to Y'$ (resp. there are $2$ sections of $Z'\to Y'$ which only coincide over $U'$).
Pick a closed point $y'$ in $Y'\setminus U'$ and a generization $u'\in U'$ such that $C:=\overline{u'}$ has dimension $1$. Let $C'=Z\cap X'\times_{Y'} C$. QED
Now we refine Lemma 1 and show that it is enough to consider either "vertical curves" (contained in a fiber) or "horizontal curves" (birational to a curve in the base).
Lemma 2. Let $f\colon X\to Y$ be a morphism of finite type between noetherian schemes. If $f$ is not universally closed (resp. not separated) then there exists a birational morphism of curves $C'\to C$ which is not universally closed (resp. not separated) such that either:
(i) there is a finite morphism $C'\to X_y$ for a point $y\in Y$ (of finite type) and $C$ is a projective curve over $y$; or
(ii) there is a finite morphism $C\to \operatorname{Spec} \mathcal{O}_{Y,y}$ and $C'$ is a closed subscheme of $X\times_Y C$.
Proof. Let $C\to Y$ be a curve as in Lemma 1. The image is either a point or the morphism is quasi-finite.
Case (i): the image of $C$ is a point $y\in Y$. We may replace $C$ by a compactification and assume that $C\to y$ is projective. If the image of $C'\to X_y$ is a point, then it is a closed point and it follows that $C'\to C$ is finite which gives a contradiction since $C'\to C$ is not proper. Thus the image of $C'\to X_y$ is $1$-dimensional and $C'\to X_y$ is quasi-finite and proper, hence finite.
Case (ii): $C\to Y$ is quasi-finite. We may replace $Y$ with the schematic image of $C$ and localize at the image of a suitable closed point of $C$. Then $Y$ is integral and $1$-dimensional. Using Zariski's main theorem, we have $C\to \overline{C}\to Y$ where $C\to \overline{C}$ is an open immersion and $\overline{C}\to Y$ is finite. We may replace $C$ with $\overline{C}$ and $C'$ with its closure in $X\times_Y \overline{C}$. QED
Remark. The localization in (ii) is only necessary when $Y$ is not Jacobson.
The third lemma shows that curves as in Lemma 2 give rise to non-coherent cohomology.
Lemma 3. Let $f\colon C'\to C$ be a birational morphism of finite type of curves. If $f$ is not universally closed (resp. not separated), then $f_*\mathcal{O}_{C'}$ is not coherent (resp. $R^1f_*\mathcal{O}_{C'}$ is not coherent). If $C$ is projective over $k$, then in addition $\Gamma(C',\mathcal{O}_{C'})$ (resp. $H^1(C',\mathcal{O}_{C'})$) is infinite-dimensional.
Proof. For the first statement, it is enough to show that the sheaf is not coherent after passing to the henselization of a suitable point $c$ in $C$. We may thus assume that $C$ is local and henselian (although not necessarily irreducible). If $f$ is not universally closed, then (after replacing $C'$ with a connected component if it is reducible) $C'\to C$ is an open immersion. Then $f_*\mathcal{O}_{C'}$ is not coherent.
If $f$ is not separated, then (after replacing $C'$ with a connected component if it is reducible) $C'\to C$ is not separated and there is an open covering of $C'$ consisting of the local rings at the points above $c$. A Cech calculation gives that $H^0(C',\mathcal{O}_{C'})$ is coherent whereas $H^1(C',\mathcal{O}_{C'})$ is not coherent.
For the second statement, we note that there is a dense open subscheme $U\subseteq C$ such that $f_*\mathcal{O}_{C'}$ is coherent over $U$ and $R^1f_*\mathcal{O}_{C'}$ is zero over $U$. If $f_*\mathcal{O}_{C'}$ is not coherent, then choose a coherent subsheaf $\mathcal{F}\subseteq f_*\mathcal{O}_X$ that is an isomorphism over $U$. Then the cokernel $\mathcal{Q}$ is a non-coherent sheaf that is zero over $U$. It follows that $H^0(C,\mathcal{Q})$ is infinite, hence that $H^0(C,f_*\mathcal{O}_{C'})$ is infinite since $H^1(C,\mathcal{F})$ is finite. If $R^1f_*\mathcal{O}_{C'}$ is not coherent, then $H^0(C,R^1f_*\mathcal{O}_{C'})$ is infinite. QED.
Theorem 2 is an easy consequence of the lemmas above:
Proof of Theorem 2. We have seen that the conditions are necessary. Conversely, assume that $f$ is not universally closed (resp. not separated). We have to show that $f_*$ does not preserve coherence (resp. $R^1f_*$ does not preserve coherence). Let $C'\to C$ be as in Lemma 2 and choose a coherent sheaf $\mathcal{F}\in \mathrm{Coh}(X)$ restricting to the push-forward of $\mathcal{O}_{C'}$ in $\mathrm{Coh}(X\times \operatorname{Spec} \mathcal{O}_{Y,y})$. By Lemma 3, we have that $f_*\mathcal{F}$ (resp. $R^1f_*\mathcal{F}$) is not coherent at $y$. QED.
Remark 1. With small modifications, the proof of Theorem 2 also holds for algebraic spaces and Deligne–Mumford stacks: first one takes a finite cover of $Y$ to reduce to $Y$ a scheme, then uses the valuative criterion for stacks. An alternative proof for schemes, algebraic spaces and Deligne–Mumford stacks would be to use Raynaud–Gruson's Chow lemma to reduce the question to morphisms of the form $f\colon X\to \mathbb{P}^n_Y \to Y$ where $f\colon X\to \mathbb{P}^n_Y$ is étale and birational and $Y$ is a scheme.
Remark 2. The situation for algebraic stacks with infinite-dimensional stabilizers is more subtle. Theorem 1 holds as mentioned above and Theorem 2 (i) is probably ok. Theorem 2 (ii) however has to be modified. If $X$ has a good moduli space then it follows from Theorem 2 that $X$ has coherent cohomology if and only if $X_\mathrm{gms}$ is proper. For example, note that $BGL_n$ has coherent cohomology but is not separated. This is all well-known and one may argue that such stacks are "almost proper" (cf. paper by Halpern-Leistner–Preygel, arXiv:1402.3204).