This works when one of $F, G$ is locally free. I am not sure whether it is true when both are merely assumed to be coherent (e.g. I don't see how to get the map). (In general, even the generalization of Serre duality -- Grothendieck duality -- tells you how to hom out of $\mathbf{R}\Gamma \mathcal{F}$ (or more generally derived push-forward) of a sheaf $\mathcal{F}$ into some complex of abelian groups, and this doesn't seem to tell you about $\mathrm{Ext}$ functors up top, in $X$, though perhaps I'm missing something).
Namely, there is a map $H^n(X, \omega) \to k$ (the "integration" map*). To get the map $$\mathrm{Ext}^i(F, G) \to \mathrm{Ext}^{n-i}(G, F \otimes \omega)^*$$ (which is natural), we need a pairing $$\mathrm{Ext}^i(F, G) \times \mathrm{Ext}^{n-i}(G, F \otimes \omega) \to k.$$ To do this, we can use the Yoneda product to pair these to $\mathrm{Ext}^n(F, F \otimes \omega)$. If $F$ is locally free, then this naturally maps to $\mathrm{Ext}^n(O_X, F \otimes F^{\vee} \otimes \omega)$, which in turns maps to $H^n(X, \omega)$ (by coevaluation) and thus to $k$. If $G$ is locally free, we can similarly write both sides as $\mathrm{Ext}^i(F \otimes G^{\vee}, \omega)$ and $\mathrm{Ext}^{n-i}(O_X, G^{\vee} \otimes F \otimes \omega)^*$, and we get the pairing and isomorphism just as in Hartshorne.
Now if we fix one of $F, G$, we get a $\delta$-functor in the other. So if the natural transformation is an isomorphism when both are locally free, it is an isomorphism when one is locally free and the other merely coherent (since on a projective scheme, every coherent sheaf has a locally free presentation, and we can use the "finite presentation trick").
*Here the comparison is as follows: on a compact complex manifold $X$ of dimension $n$, if $\omega$ denotes the sheaf of holomorphic $(n,0)$-forms, we have (Dolbeaut isomorphism) $$H^n(X, \omega) = \frac{(n,n)\mathrm{-forms}}{\overline{\partial}\mathrm{-exact\ top forms}}$$ and so we can define the map as integration, legitimately (because a $\overline{\partial}$-exact top form is exact in the usual sense, this is well-defined).