Distinguished triangle of dualizing complexes and/or determinants? Q1 : If $X \to Y \to Z$ are maps of schemes, is there a relation such as
$$\omega_{X/Z} \overset{?}{=} \omega_{Y/Z}|_X \overset{L}{\otimes} \omega_{X/Y}$$
between their dualizing complexes? Or maybe some kind of distinguished triangle?
The reason I think so is that I'm told $\omega_{X/Z}$ is the determinant $\det \mathbb L_{X/Z}$ in the sense of determinants of complexes (if $X \to Z$ is l.c.i.). The cotangent complex always sits in a distinguished triangle
$$\mathbb L_{Y/Z}|_X \to \mathbb L_{X/Z} \to \mathbb L_{X/Y},$$
which leads me to my next question:
Q2 : Does the determinant of complexes send distinguished triangles to tensor products?
Deligne emphasizes this for split exact sequences of vector spaces in La Determinante de Cohomologie: $\det (V \oplus W) = \det V \otimes \det W$ even in the sense of complexes. It's hard for me to imagine that wouldn't extend to triangles in the derived category. The stacks project 0FJW promises to add more details later, so maybe this is obvious.
This came up in a class I'm in. Let $F : X \to X^{(p)}$ be the frobenius. I think surjectivity of the map $F_* \omega_X \to \omega_{X^{(p)}}$ is one of the conditions to be "F-rational." I'd like to understand the cokernel or cone of this morphism in general to see the obstruction to F-rationality and I naively expected something like $\omega_{X/X^{(p)}} = Hom(F_* \mathcal O_X, \mathcal O_{X^{(p)}})$.
 A: A very good reference for these topics is
Lipman, Joseph: Notes on derived functors and Grothendieck duality. Foundations of Grothendieck duality for diagrams of schemes, 1–259, Lecture Notes in Math., 1960, Springer, Berlin, 2009.
For your first question, the issue is the pseudo functoriality of $(-)^!$ together with the characterization of this functor in terms of its value in the structure sheaf.
In more detail, with $f : X \to Y$, $g : Y \to Z$, $h : X \to Z$ and $h = g \circ f$. Assume that all maps are finite type separated map of noetherian schemes. In this case, we have that $h^! \cong f^! \circ g^!$ (loc. cit. Th (4.8.1)).
Second If moreover $f$ is perfect, i.e. $\mathcal{O}_X$ is relatively perfect over $Y$ then
$$
f^! \mathcal{F} \cong f^* \mathcal{F} \otimes^L f^! \mathcal{O}_Y
$$
(loc. cit. Th (4.9.4)).
By introducing the notation $\omega_f = f^! \mathcal{O}_Y$, (and similarly for $g$ and $h$) you get your desired result under the hypothesis mentioned. But beware: in full generality $\omega_f$ is a complex not concentrated in a single degree unless the morphisms are Cohen-Macaulay.
Indeed, as a consequence of the previous discussion, we have the following chain of isomorphisms
$$
\omega_h \cong h^! \mathcal{O}_Z \cong f^! g^! \mathcal{O}_Z \cong f^! \omega_g \cong
$$
$$
\cong f^* \omega_g \otimes^L f^! \mathcal{O}_Y \cong
f^* \omega_g \otimes^L \omega_f
$$
As for the formula $\omega_f \cong \det \mathbb L_{f}$, it looks plausible to me under complete intersection hypothesis. I don't know of a published proof. And I don't think it holds under more general hypothesis because without the complete intersection condition, $\mathbb L_{f}$ is not perfect.
Here I interpret $\det$ as something like $L\Lambda^n$, the derived exterior power, where $n$ denotes the relative dimension.
Finally, if $f$ is finite if follows from sheafified duality (loc. cit. Cor. (4.3.6)) that
$$
f^! \mathcal{F} \cong \mathbf{R}\mathcal{H}om(f_*\mathcal{O}_X, \mathcal{F})^{\tilde{}}
$$
If you substitute by Frobenius you get you last formula, if I understand well.
A: As for the general question about determinants and distinguished triangles, the answer is no, as stated: this is why $\infty$-categories are useful. However, it is true that the determinant is well defined in terms of $\infty$-categories and that is behaves in the way we expect. The point is that $K$-theory is a sheaf of the Zariski topology (at least if we restrict to quasi-compact and quasi-separated schemes - otherwise, we should Zariski sheafify it to make things work properly), and that as such a sheaf of (connective) spectra, and that it can be not only recovered from free modules of finite type, but from perfect complexes as well. More precisely, formula
$$\mathit{det}(E\oplus F)\simeq\mathit{det}(E)\otimes_{\mathcal{O}_X}\mathit{det}(F)$$
expresses the fact that, if we let $\mathit{Free}(X)$ be the groupoid of free $\mathcal{O}_X$-modules of finite rank
the functor $\mathit{det}:\mathit{Free}(X)\to \mathit{Pic}(X)$, taking values in a sheaf of Picard groupoid (i.e. group-like $E_\infty$-space $\simeq$ connective spectrum), is symmetric monoidal. Since $\mathit{Pic}$ is a sheaf of (Picard) groupoids (here `sheaf' is intended in the sense of $\infty$-categories), the determinant above factors through the sheafification of $\mathit{Free}(X)$, namely the groupoid $\mathit{Vect}$ of vector bundles over $X$. Passing to $K$-theory, this gives a map
$$\mathit{det}:K(\mathit{Vect}(X))\to \mathit{Pic}(X)\, .$$
If $X$ is nice enough (has an ample line bundle), Gillet-Waldhausen Theorem says that, if $\mathit{Perf}(X)$ denotes the $\infty$-category of perfect complexes, the inclusion $\mathit{Vect}(X))\subset\mathit{Perf}(X)$ induces an equivalence of ring spectra:
$$K(\mathit{Vect}(X))\cong K(\mathit{Perf}(X))\, .$$
In general, if we restrict to quasi-compact and quasi-separated schemes, the presheaf $X\mapsto K(\mathit{Perf}(X))$ turns out to be the sheafification of $X\mapsto K(\mathit{Vect}(X))$ (this is a courtesy of Thomason and Trobaugh). For general schemes, $K(\mathit{Vect}(-))$ and $K(\mathit{Perf}(-))$ have the same Zariski sheafication (since they agree on affine schemes). Finally, we thus get by Zariski descent a functorial map
$$\mathit{det}:K(\mathit{Perf}(X))\to \mathit{Pic}(X)\, .$$
By definition of $K$-theory, for any short exact sequence of perfect complexes $E'\to E\to E''$, in the sense of $\infty$-category theory, there is an isomorphism
$$\mathit{det}(E)\simeq\mathit{det}(E')\otimes_{\mathcal{O}_X}\mathit{det}(E'')\, .$$
But having a distinguished triangle is less information than having a short exact sequence: a short exact sequence is not only given by the maps $u:E'\to E$ and $v:E\to E''$ but also by an homotopy identifying $vu$ with $0$ (together with the property that $E'$ is the fiber of $v$ or, equivalently, that $E''$ the cofiber of $u$).
As a concluding remark, everything above can be done with derived (or even spectral) schemes: the only change in the proof above is that we cannot apply the theorem of Gillet-Waldhausen. We replace it by a theorem of Lurie which expresses the fact that $K(\mathit{Vect}(X))\cong K(\mathit{Perf}(X))$ for affine spectral schemes (Theorem 5 here). If we have to deal with cotangent complexes, this is useful because the functorialities are better understood in the derived context (e.g. the compatibility with pullbacks is almost automatic: the property that the cotangent complex is perfect follows from the property that the structural map is locally of finite presentation and that the relative cotangent complex is of Tor-amplitude $[-n,1]$ for some positive integer $n$). We can get classical results by restricting our attention to that case where we have sufficiently many flat maps around.
We may also observe that the description of the determinant above also gives the uniqueness of the determinant of perfect complexes through a nice universal property: in particular, if we insist on functoriality and on the fact that it is map of connective spectra from $K$-theory to the Picard stack $\mathit{Pic}$, it is completely determined by its values on free modules of finite rank and their isomorphisms.
