One usually defines a Serre functor to be a functor on a $k$-linear category $\mathcal{C}$ which has finite dimensional $Hom$s over $k$. In that case, the standard definition is that a Serre functor $S: \mathcal{C} \to \mathcal{C}$ is an additive functor so that there exists an isomorphism of bifunctors $Hom_\mathcal{C}(A, B) \to Hom_\mathcal{C}(B, SA)^*$. Of course, one needs the finite dimensionality of $Hom$s to be allowed to take the dual in the definition.
Nonetheless, when one constructs Serre functors on bounded derived categories of algebraic varieties one first constructs the dualizing complex. Dualizing complexes make perfect sense on an arbitrary Noetherian ring, see https://stacks.math.columbia.edu/tag/0A7A. In particular, they make sense as objects in derived categories of affine schemes, where Homs are infinite dimensional essentially always.
Is there a meaningful extension to the notion of a serre functor some class of triangulated categories for which Homs are not required to be finite dimensional, so that it agrees with `tensor by dualizing complex' for quasiprojective varieties over a field? By ``meaningful'' I mean a definition that only uses notions that make sense for an arbitrary triangulated category (well, maybe an arbitrary triangulated category that comes from a dg category).
