I was going through the construction of dualizing sheaf given in Hartshorne [III, 7, Lemma 7.4]. The proof apparently omits lots of details. In particular it does not mention any of the $i^* , i_*, i^{!}$ that should appear, so almost none of the things he writes are actually defined. I was trying to clean up the proof and I noticed the following more general algebraic lemma.
Lemma: Let $\mathcal C, \mathcal D$ be abelian categories and $\mathcal D$ has enough injectives. $F : \mathcal C \to \mathcal D$ and $G:\mathcal D \to \mathcal C $ are additive functors such that
(i) $F$ is left exact and sends nonzero objects to nonzero objects.
(ii) $G$ is right adjoint to $F$.
(ii) For some fixed object $P$ of $\mathcal D$, we have $(R^i (F \circ G) )(P) = 0$ for all $i < r$ for some fixed $r \in \mathbb N$.
Then it follows that for any object $A$ of $\mathcal C$, $\text{Ext}^i (F(A), P) = 0$ for $i < r$ and there is a functorial isomorphism $\text{Ext}^r (F(A), P) \cong \text{Hom}(A, (R^r (G))(P))$.
Proof: We choose an injective resolution of $P$ in $\mathcal D$, $0 \to P \to I^{\bullet}$. Define $J^{\bullet} := G(I^{\bullet})$. Since $F$ is left exact and left adjoint it is in fact exact. By adjointness, $J^{\bullet}$ is a complex of injective objects. Now $F(h^iJ^{\bullet}) = h^i( F(J^{\bullet}))= h^i (F \circ G )(I^{\bullet}) = R^i(F \circ G)= 0$ for $i<r$ by hypothesis. So $J^{\bullet}$ is exact till $r$-th step. Therefore we can write $J^{\bullet}$ as $J_1^{\bullet} \oplus J_2^{\bullet}$, where $J_1^{\bullet}$ is injective, exact and nonzero only in degrees $\le r$ and $J_2$ is also injective and nonzero only in degrees $\ge r$. Now $\text{Ext}^r(F(A), P) = h^r (\text{Hom}(F(A), I^{\bullet})= h^r(\text{Hom}(A, J^{\bullet}))$ $=h^r(\text{Hom}(A, J_2^{\bullet})) = $ $\text{Ker}((\text{Hom}(A, J_2^r) \to \text{Hom}(A, J_2^{r+1}))$ $ = \text{Hom}(A, \text{Ker}(J_2^{r} \to J_2^{r+1}))$. But $\text{Ker}(J_2^{r} \to J_2^{r+1}) = h^r(J^{\bullet})= h^r(G(I^{\bullet}))= (R^rG)(P)$.
Now to prove the existence of dualizing sheaf for a closed immersion $j : X \to \mathbb{P}^n_k$, we take $\mathcal C = \mathfrak {Qco} (X), \mathcal D = \mathfrak {Qco} (\mathbb P ^n _k), F = j_*, G = j^{!}$. Then $(R^i (j_* \circ j^{!})) (\omega_{\mathbb{P}^n_k}) = \mathscr{Ext}^i(j_* \mathscr{O}_X, \omega_{\mathbb{P}^n_k}) = 0$ for $i < r:= n - \dim X$. Therefore, it follows from lemma that $\text{Ext}^r(j_*\mathscr{F}, \omega_{\mathbb{P}^n_k}) = \text{Hom}_{X}(\mathscr{F}, (R^{r} (j^!))(\omega_{\mathbb{P}^n_k}) )$. Now $(R^{r} (j^!))(\omega_{\mathbb{P}^n_k})$ is same as the quasicoherent $\mathcal{O}_X$- module corresponding to the quasicoherent $j_* \mathcal{O}_X$ module $\mathscr{Ext}^r (j_* \mathcal O _{X}, \omega_{\mathbb{P}^n_k})$, which we write by $\mathscr{Ext}^r (j_* \mathcal O _{X}, \omega_{\mathbb{P}^n_k})^{\sim}$.
Questions:
(i)Does the lemma follow quickly from some spectral sequence (which I don't properly know) argument? Is there any reference for the lemma?
(ii)In these notes (http://ocw.mit.edu/courses/mathematics/18-726-algebraic-geometry-spring-2009/lecture-notes/MIT18_726s09_lec24_dualizing.pdf), the dualizing sheaf is defined to be $j^*(\mathscr{Ext}^r (j_* \mathcal{O}_X, \omega_{\mathbb{P}^n_k}))$. Is it the same as $\mathscr{Ext}^r (j_* \mathcal{O}_X, \omega_{\mathbb{P}^n_k})^{\sim}$?
I don't really understand some parts of the proof in that note, in particular it talks about isomorphism of sheaves in two different topological spaces, which should really be related by a push forward.