A very good reference for these topics is
Lipman, Joseph: Notes on derived functors and Grothendieck duality. Foundations of Grothendieck duality for diagrams of schemes, 1–259, Lecture Notes in Math., 1960, Springer, Berlin, 2009.
For your first question, the issue is the pseudo functoriality of $(-)^!$ together with the characterization of this functor in terms of its value in the structure sheaf.
In more detail, with $f : X \to Y$, $g : Y \to Z$, $h : X \to Z$ and $h = g \circ f$. Assume that all maps are finite type separated map of noetherian schemes. In this case, we have that $h^! \cong f^! \circ g^!$ (loc. cit. Th (4.8.1)).
Second If moreover $f$ is perfect, i.e. $\mathcal{O}_X$ is relatively perfect over $Y$ then $$ f^! \mathcal{F} \cong f^* \mathcal{F} \otimes^L f^! \mathcal{O}_Y $$ (loc. cit. Th (4.9.4)). By introducing the notation $\omega_f = f^! \mathcal{O}_Y$, (and similarly for $g$ and $h$) you get your desired result under the hypothesis mentioned. But beware: in full generality $\omega_f$ is a complex not concentrated in a single degree unless the morphisms are Cohen-Macaulay.
Indeed, as a consequence of the previous discussion, we have the following chain of isomorphisms $$ \omega_h \cong h^! \mathcal{O}_Z \cong f^! g^! \mathcal{O}_Z \cong f^! \omega_g \cong $$ $$ \cong f^* \omega_g \otimes^L f^! \mathcal{O}_Y \cong f^* \omega_g \otimes^L \omega_f $$
As for the formula $\omega_f \cong \det \mathbb L_{f}$, it looks plausible to me under complete intersection hypothesis. I don't know of a published proof. And I don't think it holds under more general hypothesis because without the complete intersection condition, $\mathbb L_{f}$ is not perfect.
Here I interpret $\det$ as something like $L\Lambda^n$, the derived exterior power, where $n$ denotes the relative dimension.
Finally, if $f$ is finite if follows from sheafified duality (loc. cit. Cor. (4.3.6)) that $$ f^! \mathcal{F} \cong \mathbf{R}\mathcal{H}om(f_*\mathcal{O}_X, \mathcal{F})^{\tilde{}} $$ If you substitute by Frobenius you get you last formula, if I understand well.