This of course can be found in any reference on duality (e.g. Hartshorne "Algebraic Geometry" chap. 3) but in Hartshorne's proof it's somehow mysterious where the $\mathscr{E}xt^{n-1}_P(\mathscr{O}, \omega_{P^n})$ comes from. You can explain it by studying relative duality for morphisms (e.g. Hartshorne "Residues and Duality"), in particular for the closed immersion $i:C\to \mathbb{P}^n$. Here is another argument (in a way it's the proof from Hartshorne's AG done backwards, so that we can see how this $\mathscr{E}xt^{n-1}$ appears).
Recall that a dualizing sheaf is a sheaf $\omega$ which satisfies $H^1(C, F)^\vee = Hom_C(F, \omega)$ for every coherent $F$ on $C$.
A thing to start with is duality on $P:=\mathbb{P}^n$: $$ H^1(C, F)^\vee = H^1(P, F)^\vee = Ext^{n-1}_P(F, \omega_P). $$
We now need a natural isomorphism between $Ext^i_P(F, \omega_P)$ and $Hom_C(F, \omega)$ for some $\omega$. A good thing to do to compare $Hom$'s and $Ext$'s on $P$ and $C$ is to find a spectral sequence comparing the two. Let us look at the functors $$ Coh(P) \to Coh(C) \to Ab $$ where the first functor is $\mathscr{H}om_P(\mathscr{O}_C, -)$ and the second is $Hom_C(F, -)$, their composition $Hom_P(F, -)$. The spectral sequence of this composition is $$ E^{ij}_2 = Ext^i_C(F, \mathscr{E}xt^j_P(\mathscr{O}_X, G)) \Rightarrow Ext^{i+j}_P(F, G). $$ For $i+j=n-1$ and $G=\omega_P$ we have our group $Ext^{n-1}_P(F, \omega)$ on the right hand side, and we have $E^{0,n-1}_2 = Hom_C(F, \mathscr{E}xt^{n-1}_P(\mathscr{O}_X, \omega_P)$, which suggests we should take $\omega = \mathscr{E}xt^{n-1}_P(\mathscr{O}_X, \omega_P)$.
To prove that this works, we need to show that $E^{ij}_2 = 0$ for $j<n-1$, which is done in Hartshorne "Algebraic Geometry", chap. 3.