A very good reference for these topics is Lipman, Joseph: Notes on derived functors and Grothendieck duality. *Foundations of Grothendieck duality for diagrams of schemes*, 1–259, Lecture Notes in Math., **1960**, Springer, Berlin, 2009. For your first question, the issue is the pseudo functoriality of $(-)^!$ together with the characterization of this functor in terms of its value in the structure sheaf. In more detail, with $f : X \to Y$, $g : Y \to Z$, $h : X \to Z$ and $h = g \circ f$. Assume that all maps are *finite type separated map of noetherian schemes*. In this case, we have that $h^! \cong f^! \circ g^!$ (loc. cit. Th (4.8.1)). Second If moreover $f$ is perfect, i.e. $\mathcal{O}_X$ is relatively perfect over $Y$ then $$ f^! \mathcal{F} \cong f^* \mathcal{F} \otimes^L f^! \mathcal{O}_Y $$ (loc. cit. Th (4.9.4)). By introducing the notation $\omega_f = f^! \mathcal{O}_Y$, (and similarly for $g$ and $h$) you get your desired result under the hypothesis mentioned. But beware: in full generality $\omega_f$ is a complex not concentrated in a single degree unless the morphisms are Cohen-Macaulay. Indeed, as a consequence of the previous discussion, we have the following chain of isomorphisms $$ \omega_h \cong h^! \mathcal{O}_Z \cong f^! g^! \mathcal{O}_Z \cong f^! \omega_g \cong $$ $$ \cong f^* \omega_g \otimes^L f^! \mathcal{O}_Y \cong f^* \omega_g \otimes^L \omega_f $$ As for the formula $\omega_f \cong \det \mathbb L_{f}$, it looks plausible to me under complete intersection hypothesis. I don't know of a published proof. And I don't think it holds under more general hypothesis because without the complete intersection condition, $\mathbb L_{f}$ is not perfect. Here I interpret $\det$ as something like $L\Lambda^n$, the derived exterior power, where $n$ denotes the relative dimension. Finally, if $f$ is finite if follows from sheafified duality (loc. cit. Cor. (4.3.6)) that $$ f^! \mathcal{F} \cong \mathbf{R}\mathcal{H}om(f_*\mathcal{O}_X, \mathcal{F})^{\tilde{}} $$ If you substitute by Frobenius you get you last formula, if I understand well.