Let $X$ be a CohenMacaulay variety. $\omega$ denotes the dualizing sheaf of $X$. Can we get $\mathcal{Hom}(O_X,\omega)\otimes \mathcal{Hom}(\omega, O_X)=O_X$?

7$\begingroup$ No. This is equivalent to saying that $\omega$ is invertible, which in turn is equivalent to $X$ being Gorenstein. $\endgroup$ – BCnrd Dec 3 '10 at 6:49

1$\begingroup$ $\mathcal{Hom}(\omega, \omega)=O_X$ ? $\endgroup$ – Diego Maradona Dec 3 '10 at 7:55

1$\begingroup$ As BCnrd says, this is equivalent to $\omega$ being invertible. There are two plausible maps you could use at the level of (local) rings: $M \otimes_R \mathrm{Hom}_R(M,R) \to \mathrm{End}_R(M)$ sending $x \otimes f$ to $y \mapsto f(y)x$, or $M \otimes_R \mathrm{Hom}_R(M,R) \to R$ sending $x \otimes f$ to $f(x)$. In either case, for $\omega$ the image is the order ideal of $\omega$, the ideal of $R$ generated by images of $\omega$. If that's all of $R$ then $\omega$ is free. $\endgroup$ – Graham Leuschke Dec 3 '10 at 12:06
The answer is no. $\mathcal Hom (\mathcal O_X,\omega)\simeq \omega$, but $\omega\otimes \mathcal Hom (\omega, \mathcal O_X)$ is not necessarily reflexive, let alone locally free. However, it is true that if $X$ is $G_1$, that is, Gorenstein in codimension $1$, then $$ (\omega\otimes \mathcal Hom (\omega, \mathcal O_X))^{**}\simeq \mathcal O_X. $$
EDIT: (due to popular demand, here is a better formed statement for the CM is not even needed part of the original)
This is actually true in a little more general setting: For simplicity assume that $X$ is $G_1$, $S_2$, equidimensional of dimension $d$ and admits a dualizing complex denoted by $\omega_X^\bullet$. (If, say, by variety you mean a quasiprojective (reduced) scheme of finite type over a field, then the last assumption is automatic. If in addition you also mean irreducible, then so is the equidimensionality. CM obviously implies $S_2$.)
Let $$\omega_X := h^{d}(\omega_X^\bullet)$$ and $$\omega_X^*:=\mathcal Hom_X (\omega_X, \mathcal O_X).$$
Then $${(\omega_X\otimes \omega_X^*)}^{**}\simeq \mathcal O_X.$$
This follows by the fact that $X$ is $S_2$, both sides are reflexive and they agree in codimension $1$ due to the $G_1$ assumption.
EDIT2: (inspired by Karl's answer): This actually also implies that $$\mathcal Hom_X(\omega_X,\omega_X)\simeq \mathcal O_X$$ (under the same conditions) since on the open set where $\omega_X$ is a line bundle, $$\mathcal Hom_X(\omega_X,\omega_X)\simeq {\omega_X\otimes \omega_X^*}$$ and then since they are both reflexive and $X$ is $S_2$, $$\mathcal Hom_X(\omega_X,\omega_X)\simeq ({\omega_X\otimes \omega_X^*})^{**}\simeq \mathcal O_X.$$

2$\begingroup$ Dear Sandor: if $X$ isn't CM, so the dualizing complex isn't concentrated in a single degree (on each connected component), then what is meant by $\omega$? Its top nonzero homology sheaf? $\endgroup$ – BCnrd Dec 3 '10 at 7:09

1$\begingroup$ Perhaps Sandor meant the canonical sheaf, which can be defined as long as $X$ is embeddable in a Gorenstein scheme? $\endgroup$ – Hailong Dao Dec 3 '10 at 14:00

1$\begingroup$ Sandor was certainly working on the geometric setting (ie, variety) where dualizing complexes exist, and then by $\omega$ he meant the first nonzero sheaf as BCnrd suggested. This G1 + S2 hypothesis is pretty common when one wants to work on nonnormal varieties but still have a ``canonical divisor''. $\endgroup$ – Karl Schwede Dec 3 '10 at 15:12

1$\begingroup$ Dear Brian: you are right (as always). I wrote that in haste and as Hailong and Karl correctly guessed I was thinking of the canonical sheaf in case $X$ is not CM. Of course, it would be wrong to call that the dualizing sheaf. Although, I guess one might consider that in that case nothing deserves that name, so the abuse was not extremely bad (just simply bad). :) $\endgroup$ – Sándor Kovács Dec 3 '10 at 16:41

$\begingroup$ Sandor, that's a good point. The argument that $\mathcal{H}om_X(\omega_X, \omega_X) \cong \mathcal{O}_X$ is substantially easier than what I wrote when $X$ is G1 (which implies that $\omega_X$ is a line bundle on a big open set). $\endgroup$ – Karl Schwede Dec 4 '10 at 3:46
Above, in a comment, Hao Sun asked if it was true that $\mathcal{H}om_X(\omega, \omega) = \mathcal{O}_X$. Here I will assume that $X$ is reduced and irreducible and $\omega$ is the canonical sheaf as discussed in S\'andor's answer.
This second question is true if and only if the variety in question is S2 (so in particular, it holds in the CohenMacaulay case).
In fact, $\mathcal{H}om_X(\omega, \omega)$ is the S2ification of $\mathcal{O}_X$ (and thus, Spec of it is the S2ification of $X$).
EDIT: A reference for this last fact is Aoyama, "Some basic results on canonical modules", also see Aoyama, "On the depth and projective dimension of the canonical module".
EDIT2: Let me also sketch an idea for why this last statement is true. On a nonS2 variety $X$, the canonical sheaf of $X$ is the same as the canonical sheaf of the S2ification of $X$ (up to pushdown). To see this, observe that S2ification is an operation outside a set of codimension 2, and also observe that $\omega$ itself is always an S2sheaf for a variety (see for example, a paper of Hartshorne, "Generalized divisors and biliaison"). It then quickly follows that $$ \mathcal{H}om_{O_X}(\omega,\omega) = \mathcal{H}om_{O_{X^{S2}} }(\omega, \omega).$$
This sheaf is clearly S2 (since $\omega$ is), furthermore, $$R\mathcal{H}om_{O_{X^{S2}} }^{.}(\omega^{.}, \omega^{.}) \cong {\mathcal O_{X^{S2}}}.$$ All the other terms appearing in that spectral sequences used to compute that have support at a codimension2 subset, and it follows that ${\mathcal O_{X^{S2}}}$ and $\mathcal{H}om_{O_{X^{S2}} }(\omega, \omega)$ are isomorphic outside a set of codimension 2, and the result follows.
However if $X$ is proper and finite type over any field there does exist a dualizing sheaf, and it agrees with the top nonzero homology sheaf of the dualizing complex.
Addendum
I mean by dualizing sheaf, a sheaf that represents the functor $\mathcal{F} \mapsto H^n(X,\mathcal{F})^{\vee}$, i.e. such that there is an ismorphism
$$ \mathrm{Hom}_X(\mathcal{F},\omega_X) \cong H^n(X,\mathcal{F})^{\vee}. $$
For me, canonical is related to local duality, in this sense $\omega_{X,x}$ is a canonical module for the ring $\mathcal{O}_{X,x}$.
Of course you need Cohen Macaulay to have
$$ \mathrm{Ext}^i_X(\mathcal{F},\omega_X) \cong H^{ni}(X,\mathcal{F})^{\vee}. $$ for every $i$; this would make $\omega_X$ a dualizing complex. This story is explained in a somewhat elementary way (without derived categories) in Kleiman's "Relative duality for quasicoherent sheaves" in Compositio Mathematica. In the case that $k$ is a perfect field there is characterization of $\omega_X$ in terms of differentials and traces, see Lipman's Asterisque 117, aka "Lipman's blue book". In this book a nice treatment of residues is given in this setting.

$\begingroup$ Leo, based on the above discussion, would it be better to call that sheaf the "canonical sheaf" (or were you assuming your variety was CM). Also, I don't think the perfect assumption is needed. $\endgroup$ – Karl Schwede Dec 3 '10 at 17:43

1$\begingroup$ Karl, thank you for your comment, I have tried to be more explicit with this correction. $\endgroup$ – Leo Alonso Dec 4 '10 at 22:03