A very good reference for these topics is 

Lipman, Joseph: Notes on derived functors and Grothendieck duality. *Foundations of Grothendieck duality for diagrams of schemes*, 1–259, Lecture Notes in Math., **1960**, Springer, Berlin, 2009.

For your first question, the issue is the pseudo functoriality of $(-)^!$ together with the characterization of this functor in terms of its value in the structure sheaf. 

In more detail, with $f : X \to Y$, $g : Y \to Z$, $h : X \to Z$ and $h = g \circ f$. Assume that all maps are *finite type separated map of noetherian schemes*. In this case, we have that $h^! \cong f^! \circ g^!$ (loc. cit. Th (4.8.1)).

Second If moreover $f$ is perfect, i.e. $\mathcal{O}_X$ is relatively perfect over $Y$ then
$$
f^! \mathcal{F} \cong f^* \mathcal{F} \otimes^L f^! \mathcal{O}_Y
$$
(loc. cit. Th (4.9.4)).
By introducing the notation $\omega_f = f^! \mathcal{O}_Y$, (and similarly for $g$ and $h$) you get your desired result under the hypothesis mentioned. But beware: in full generality $\omega_f$ is a complex not concentrated in a single degree unless the morphisms are Cohen-Macaulay.

Indeed, as a consequence of the previous discussion, we have the following chain of isomorphisms
$$
\omega_h \cong h^! \mathcal{O}_Z \cong f^! g^! \mathcal{O}_Z \cong f^! \omega_g \cong
$$
$$
\cong f^* \omega_g \otimes^L f^! \mathcal{O}_Y \cong
f^* \omega_g \otimes^L \omega_f
$$

As for the formula $\omega_f \cong \det \mathbb L_{f}$, it looks plausible to me under complete intersection hypothesis. I don't know of a published proof. And I don't think it holds under more general hypothesis because without the complete intersection condition, $\mathbb L_{f}$ is not perfect.

Here I interpret $\det$ as something like $L\Lambda^n$, the derived exterior power, where $n$ denotes the relative dimension.

Finally, if $f$ is finite if follows from sheafified duality (loc. cit. Cor. (4.3.6)) that
$$
f^! \mathcal{F} \cong \mathbf{R}\mathcal{H}om(f_*\mathcal{O}_X, \mathcal{F})^{\tilde{}}
$$
If you substitute by Frobenius you get you last formula, if I understand well.