Serre duality in families

Question

In Ravi Vakil's lecture notes ("Foundations of Algebraic Geometry", Classes 53 and 54) one can find a relative version of Serre duality (Exercise 6.1), namely:

"Suppose $\pi: X\rightarrow Y$ is a flat projective morphism of locally Noetherian schemes, of relative dimension $n$. Assume all of the geometric fibers are Cohen-Macaulay. Then there exists a coherent sheaf $\omega_{X/Y}$ on $X$, along with a trace map $R^n\pi_\ast\omega_{X/Y}\rightarrow\mathcal{O} _Y$ such that, for every finite rank locally free sheaves $\mathcal{F}$ on $X$, each of whose higher pushforwards are locally free on $Y$, $$R^i\pi_\ast\mathcal{F}\times R^{n-i}\pi_\ast(\mathcal{F^\vee\otimes\omega}_X)\rightarrow R^n\pi_\ast\mathcal{\omega}_X\rightarrow\mathcal{O}_Y$$ is a perfect pairing."

For citing purposes, I'd like to have a more canonical reference (i.e. paper or textbook) of this result, but was yet unable to find any. Moreover, I'd actually like to have that result for a flat proper morphism instead of a flat projective morphism. Is it also true in this case?

I'm sorry if this question is trivial, I'm not really familiar with algebraic geometry. Thank you!

Answer 1

In this case, Serre duality in families = Grothendieck duality. At the level of generality you are asking, I suggest the paper:

Kleiman, Steven L.: Relative duality for quasicoherent sheaves. Compositio Math. 41 (1980), no. 1, 39–60.

http://www.numdam.org/item/?id=CM_1980__41_1_39_0

But if you are seriously interested in the topic you should get into the derived category formulation. A modern exposition is in

Lipman, Joseph; Hashimoto, Mitsuyasu: Foundations of Grothendieck duality for diagrams of schemes. Lecture Notes in Mathematics, 1960. Springer-Verlag, Berlin, 2009.

Lipman's contribution is available also at:

https://www.math.purdue.edu/~lipman/Duality.pdf