Is it true that a line bundle is relatively ample iff its restsriction to fibers is? If so, what would be the reference?

15$\begingroup$ Probably you want the morphism in question to be proper, and also finitely presented (if no noetherian hypotheses). If so, then one can say a bit more: the locus of points in the base for which the line bundle is ample on fibers is open, and over that open in the base it is relatively ample. See EGA IV$_3$, 9.6.4. If you insist on assuming the base scheme is locally noetherian then simpler proofs of these can be found in EGA III$_1$, 4.7.1. Could also ask if true for proper morphisms not assumed to be finitely presented. I have some ideas on that, but not clear (and probably nobody cares). $\endgroup$ – BCnrd Jun 12 '10 at 22:35

$\begingroup$ If the morphism is projective, you can see this using the relative Kleiman's criterion. $\endgroup$ – Henri Jun 12 '10 at 22:49

$\begingroup$ @BCnrd Do you mean EGA $III_1$, 4.7.1 in Chapter III page 434? But I am not able to find the result you referred to... $\endgroup$ – Li Yutong Aug 10 '17 at 19:36
If you admit the map to be proper and the schemes to be reasonably good it is true. A reference I know is Lazarsfeld's book "Positivity in algebraic geometry", paragraph 1.7.

1$\begingroup$ Thank you. Actually, the previous comment, which had to be called an answer, settles the question via a reference to EGA. $\endgroup$ – Roman Fedorov Jun 13 '10 at 15:48
EDIT : my previous answer was wrong. Thanks to BConrad for pointing it out.
Here is a counterexample if the map is not proper. Let $X$ be the plane, let $Y$ be the blowup of the plane in one point, and $U$ be $Y$ with one point of the exceptionnal divisor removed. Let $f_U:U\to X$ be the projection and consider the line bundle $\mathcal{O}_U$.
Since the fibers of $f_U$ are affine (either points or the affine line), $\mathcal{O}_U$ becomes ample when restricted to fibers of $f_U$. However, $\mathcal{O}_U$ is not $f_U$ample. Indeed, if it were, $\mathcal{O}_U$ would be ample on $U$, but we can compute : $$H^0(U,\mathcal{O}_U^N)=H^0(U,\mathcal{O}_U)=H^0(Y,\mathcal{O}_Y)=H^0(X,\mathcal{O}_X),$$
where the second equality comes from property $S2$ and the third holds because $f_* \mathcal{O}_Y=\mathcal{O}_X$. Hence, $H^0(U,\mathcal{O}_U^N)$ cannot distinguish between two points of the exceptionnal divisor, and $\mathcal{O}_U$ cannot be ample on $U$.
Warning : what follows is false. I kept it here so that the comment below remains understandable.
Here is a counterexample if the map is not proper. Consider the inclusion $f$ of the plane minus a point $U$ in the plane $X$. The line bundle $\mathcal{O}_U$ is ample restricted to the fibers of $f$ (they're points...). However, it is not $f$ample. Indeed, if it were, $\mathcal{O}_U$ would be ample on $U$. Choosing $N>>0$, we would get $$H^1(U,\mathcal{O}_U)=H^1(U,\mathcal{O}_U^N)=0.$$ But a simple computation via Cech cohomology shows that this cohomology group is not trivial (in fact, infinite).

3$\begingroup$ What is your definition of "ample"? For the definition in EGA (which is for quasicompact separated schemes, or in the relative case qc and separated morphisms), the cohomological criterion only applies in the proper case. Following the definition of ample in EGA II 4.5 and of quasiaffine in EGA II 5.1, one has that qaffine is equivalent to ampleness of the structure sheaf. So the above counterexample is not a counterexample if using the EGA definition of ample. $\endgroup$ – BCnrd Jun 13 '10 at 12:47

$\begingroup$ Sorry, and thank you for your comment : you're right ! I edited my answer to give a genuine counterexample. In fact, in view of Zariski's Main Theorem, if I'm not mistaken, there will be no quasifinite (and separated) counterexample. $\endgroup$ – Olivier Benoist Jun 13 '10 at 20:05