MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).
show/hide this revision's text 2 added 1268 characters in body; added 191 characters in body

This works directly when one of $F, G$ is locally free. I am not sure whether it is true when both are merely assumed to be coherent (e.g. I don't see how to get the map). (In general, even the generalization of Serre duality -- Grothendieck duality -- tells you how to hom out of $\mathbf{R}\Gamma \mathcal{F}$ (or more generally derived push-forward) of a sheaf $\mathcal{F}$ into some complex of abelian groups, and this doesn't seem to tell you about $\mathrm{Ext}$ functors up top, in $X$, though perhaps I'm missing something). See below for the extension without local freeness hypotheses.

*Here the comparison is as follows: on a compact complex manifold $X$ of dimension $n$, if $\omega$ denotes the sheaf of holomorphic $(n,0)$-forms, we have (Dolbeaut isomorphism) $$H^n(X, \omega) = \frac{(n,n)\mathrm{-forms}}{\overline{\partial}\mathrm{-exact\ top forms}}$$ and so we can define the map as integration, legitimately (because a $\overline{\partial}$-exact top form is exact in the usual sense, this is well-defined).

Edit: As above, the obstacle to defining the map was that there was no natural trace morphism $$\mathrm{Ext}^n(F, F \otimes \omega) \to H^n(X, \omega)$; given one, the same arguments would answer your question for the case of $F, G$ both only assumed coherent. As Donu Arapura observes below, there is a natural way to define the trace. Reason: $F$ can be replaced by a bounded complex of locally frees in the derived category (it is a "perfect" complex) since we are working over a smooth variety (in particular, this means that any locally free resolution can be truncated at a finite stage to still yield a locally free one, by Serre's theorem on the finiteness of global dimension). For a bounded complex of locally frees $K^\bullet$, we can define a map$$\mathrm{Ext}^n(K^\bullet, K^\bullet \otimes \omega) \to H^n(X, \omega)$$by taking the "partial trace." One can think of the former as consisting of maps $K^\bullet \to K^\bullet \otimes \omega[n]$, or $\mathbf{R}\underline{Hom}(K^\bullet, K^\bullet) \to \omega[n]$. (By the conditions on $K^\bullet$, the derived internal hom is the same as the usual sheaf hom.) Since there is a natural map from $\mathcal{O}_X$ to the derived internal hom (given by the identity), we can define the trace. To show that map you are interested in becomes an isomorphism, we use the same "finite presentation" trick.
show/hide this revision's text 1

This works when one of $F, G$ is locally free. I am not sure whether it is true when both are merely assumed to be coherent (e.g. I don't see how to get the map). (In general, even the generalization of Serre duality -- Grothendieck duality -- tells you how to hom out of $\mathbf{R}\Gamma \mathcal{F}$ (or more generally derived push-forward) of a sheaf $\mathcal{F}$ into some complex of abelian groups, and this doesn't seem to tell you about $\mathrm{Ext}$ functors up top, in $X$, though perhaps I'm missing something).

Namely, there is a map $H^n(X, \omega) \to k$ (the "integration" map*). To get the map $$\mathrm{Ext}^i(F, G) \to \mathrm{Ext}^{n-i}(G, F \otimes \omega)^*$$ (which is natural), we need a pairing $$\mathrm{Ext}^i(F, G) \times \mathrm{Ext}^{n-i}(G, F \otimes \omega) \to k.$$ To do this, we can use the Yoneda product to pair these to $\mathrm{Ext}^n(F, F \otimes \omega)$. If $F$ is locally free, then this naturally maps to $\mathrm{Ext}^n(O_X, F \otimes F^{\vee} \otimes \omega)$, which in turns maps to $H^n(X, \omega)$ (by coevaluation) and thus to $k$. If $G$ is locally free, we can similarly write both sides as $\mathrm{Ext}^i(F \otimes G^{\vee}, \omega)$ and $\mathrm{Ext}^{n-i}(O_X, G^{\vee} \otimes F \otimes \omega)^*$, and we get the pairing and isomorphism just as in Hartshorne.

Now if we fix one of $F, G$, we get a $\delta$-functor in the other. So if the natural transformation is an isomorphism when both are locally free, it is an isomorphism when one is locally free and the other merely coherent (since on a projective scheme, every coherent sheaf has a locally free presentation, and we can use the "finite presentation trick").

*Here the comparison is as follows: on a compact complex manifold $X$ of dimension $n$, if $\omega$ denotes the sheaf of holomorphic $(n,0)$-forms, we have (Dolbeaut isomorphism) $$H^n(X, \omega) = \frac{(n,n)\mathrm{-forms}}{\overline{\partial}\mathrm{-exact\ top forms}}$$ and so we can define the map as integration, legitimately (because a $\overline{\partial}$-exact top form is exact in the usual sense, this is well-defined).