Let $X$ be a normal Gorenstein complex surface with $H^i(X,\mathcal{O}_X)=0$ for $i>0$ and $F$ be a rank one reflexive sheaf on $X$. I'm trying to find some ways to determine local Ext $\mathcal{E}xt^i_X(F,F)$.
For $i=0$, by the normality of $F$, I think we have $\mathcal{H}om_X(F,F)=\mathcal{O}_X$. Is there any similar result for $i>0$ (e.g. $\mathcal{E}xt^i_X(F,F)=0$ for $i>0$)?
The only thing now I can show is that $\mathcal{E}xt^i_X(F,F)$ is supported on points for $i>0$.
Firstly, note that formation of $\mathscr Hom_{\mathcal O_X}(\mathscr F,\mathscr F)$ is local, so we don't need global assumptions such as $H^i(X,\mathcal O_X) = 0$ for $i > 0$.
Secondly, I agree that the natural map $\mathcal O_X \to \mathscr Hom_{\mathcal O_X}(\mathscr F,\mathscr F)$ is an isomorphism whenever $\mathscr F$ is a rank $1$ reflexive sheaf on a normal (integral) Noetherian scheme $X$. Indeed, there exists an open $j \colon U \hookrightarrow X$ with $\operatorname{codim}(X\setminus U,X) \geq 2$ such that $j^*\mathscr F$ is locally free of rank $1$ and the unit $\mathscr F \to j_*j^*\mathscr F$ of the adjunction $j^* \dashv j_*$ is an isomorphism; see [Tag 0AY6]. Then the natural map $$\mathcal O_U \to \mathscr Hom_{\mathcal O_U}(j^*\mathscr F,j^*\mathscr F) = j^*\mathscr Hom_{\mathcal O_X}(\mathscr F,\mathscr F)$$ is an isomorphism, so the claim follows from [Tags 0AY4 and 0EBJ].
But the higher vanishing does not hold:
Example. Let $k$ be a field, and let $R = k[x,y,z]/(z^2-xy)$ be the quadratic cone. Let $I = (x,z)$ be the ideal of the $y$-axis. We get a surjection $R^2 \to I$ by $(a,b) \mapsto ax-bz$, and write $K \subseteq R^2$ for the kernel. Then the second projection $\pi \colon K \to R$ is injective with image $I$: if $(a,0) \in K$, then $a = 0$ since $R$ is a domain, showing injectivity. The image of $\pi \colon K \to R$ is those $r \in R$ such that $rz \in (x)$, i.e. the annihilator of $z$ in $R/(x) \cong k[x,y,z]/(x,z^2)$. This is the ideal $I = (x,z)$. (Concretely, $K$ is generated by $(z,x)$ and $(y,z)$.) Thus we get a short exact sequence $$0 \to I \to R^2 \to I \to 0,$$ which shows that $I$ is reflexive [Tag 0AV2] (see also [Tag 0EBM]). Applying $\operatorname{Hom}_R(-,I)$ gives an exact sequence $$0 \to \operatorname{Hom}_R(I,I) \to \operatorname{Hom}_R(R^2,I) \to \operatorname{Hom}_R(I,I) \to \operatorname{Ext}^1_R(I,I) \to 0.$$ By the above, the first terms read $$0 \to R \to I^2 \to R.$$ The final map cannot be surjective, for then $I$ would be projective: if $M$ is any $R$-module, then the short exact sequence $0 \to R \to I^2 \to R \to 0$ would give $$0 = \operatorname{Ext}^i_R(R,M) \to \operatorname{Ext}^i_R(I^2,M) \to \operatorname{Ext}^i_R(R,M) = 0,$$ showing that $\operatorname{Ext}^i_R(I,M)^2 = 0$ for $i > 0$. This is absurd since $I$ is not a principal ideal. We conclude that $\operatorname{Ext}^1_R(I,I)$ cannot be zero. $\square$
Perhaps I can say that finitely generated modules for which
$$R \to \mathrm{Hom}_R(M, M) \;\;\;\text{is an isomorphism, where $1 \mapsto id$}$$
and
$$\mathrm{Ext}^i_R(M,M) = 0 \;\;\;\text{ for $i > 0$}$$
are called semi-dualizing modules (if they are in $D^b_{coh}(R)$, they are called semi-dualizing complexes).
This can be more compactly written as
$$R \to {\bf R}\mathrm{Hom}_R(M, M)$$
is an isomorphism.
The point is that $M$ would be a dualizing/canonical module (respectively complex) if it had finite injective dimension.
There's actually a lot of work on identifying semi-dualizing modules/complexes, and they are relatively rare. For a Gorenstein local ring $R$, the only semi-dualizing module is $R$ itself up to isomorphism (in fact, the only semi-dualizing complex is $R$ up to shift). See Corollary 8.6 in Christensen, Semi-dualizing complexes and their Auslander categories. For a Gorenstein variety, it follows that the only semi-dualizing modules are line bundles.
If you weaken the condition of Gorenstein to Cohen-Macaulay, then there can be more. If I recall correctly, there are still only finitely many, and there's an even number (see Christensen-Wagstaff). In fact, unless things have changed in the past few years, no one knows an example where the number of semi-dualizing modules/complexes is not $2^n$ for some $n$.
I also found this somewhat older survey of semi-dualizing modules by Sather-Wagstaff: Semidualizing modules.
