Recall that Serre duality says that $H^{0}(X,F)\cong H^{1}(X, K\otimes F^{*})^{*}$. So the determinant of cohomology just follows from this. However we do need slightly more machinery than Serre duality, as the determinant of cohomology is defined in a relative setting ($X\rightarrow S$). In our case $S$ is affine, so Hartshorne Proposition 8.5 is suffice, which says if $X$ is Noetherian and $Y$ affine, then $R^{i}f_{*}(\mathcal{F})\cong \widetilde{H^{i}(X,\mathcal{F})}$ for any quasi-coherent $\mathcal{F}$ on $X$. Together with Serre duality this then gives you back what you wanted.
A good reference on determinant of cohomology is
Arbarello/Cornalba/Griffiths: "Geometry of algebraic curves II"
Chapter 8, which was mentioned by Robert Wilms already in his answer.