$\DeclareMathOperator\Hom{Hom}$Let $\mathcal{A}$ be an abelian category, my question is about the case when $\mathcal{A}$ is the category of quasi-coherent sheaves on a scheme $X$. There is a fully faithful embedding of $\mathcal{A}$ into the derived category $\mathcal{D}(\mathcal{A})$ given by sending an object $A\to A[0]$, a complex centred in degree zero. The inverse (defined on complexes quasi-isomorphic to those centered in degree zero) is given by $H^0$.

Now in particular, if I am not mistaken this means that for $A,B\in \mathcal{A}$ there is an isomorphism in $\mathcal{D}(\mathcal{A})$

$$\Hom_{\mathcal{A}}(A, B)[0]\cong \Hom_{\mathcal{D}(\mathcal{A})}(A[0], B[0])$$

Now to my confusion, suppose that $X$ is a normal projective Cohen–Macaulay scheme with dualizing complex $K^{\bullet}=\omega_X[d_X]$ where $\omega_X$ is a dualising sheaf. Let $A$ and $A'$ be non isomorphic sheaves such that $$\Hom_{\mathcal{A}}(A, \omega_X)\cong \Hom_{\mathcal{A}}(A', \omega_X),$$ for example this happens if $A=\Omega^i_X$ is not reflexive and $A'=A^{\vee \vee}$ is the double dual. ($\omega_X$ is in this context reflexive and can be taken to be $(\Omega^{n}_X)^{\vee \vee}$)

We then have $$\Hom_{\mathcal{A}}(\Hom_{\mathcal{A}}(A, \omega_X),\omega_X)[0]\cong \Hom_{\mathcal{A}}(\Hom_{\mathcal{A}}(A', \omega_X),\omega_X)[0]$$

Which by the argument above implies $$\Hom_{\mathcal{D}(\mathcal{A})}(\Hom_{\mathcal{D}(\mathcal{A})}(A[0], \omega_X[d_X]),\omega_X[d_X])\cong \Hom_{\mathcal{D}(\mathcal{A})}(\Hom_{\mathcal{D}(\mathcal{A})}(A'[0],\omega_X[d_X] ),\omega_X[d_X])$$

Which since $\omega_X[d_X]=K^{\bullet}$ is a dualizing complex gives

$A[0]\cong A'[0]$ in the derived category.

This seems to be a contradiction. What went wrong?

**Update:** Duality theory says that

$$R\Hom(R\Hom(A[0], \omega_X[d_X]),\omega_X[d_X])\cong A[0]$$ and $$\Hom_{\mathcal{D}(\mathcal{A})}=H^0R\Hom$$ therefore the above question should be changed into: $A$ and $A'$ are not isomorphic, but

$$\Hom_{\mathcal{A}}(A, \omega_X)\cong \Hom_{\mathcal{A}}(A', \omega_X)$$

then why does this not imply that the following objects are isomorphic in the derived category

$$A[0]\cong R\Hom(R\Hom(A[0], \omega_X[d_X]),\omega_X[d_X])\cong R\Hom(R\Hom(A'[0], \omega_X[d_X]),\omega_X[d_X])\cong A'[0]$$

Homological Algebraby Gelfand and Manin seems appropriate as far as I can deduce from your questions. $\endgroup$