Let $X$ be a locally ringed space (or a scheme) and $M,N$ two $\mathcal{O}_X$-modules such that $M \otimes N \cong \mathcal{O}_X$. Does it follow that $M$ is invertible in the usual sense, namely that $M$ is locally free of rank $1$?
It is true if $M$ is locally of finite type (which is, of course, also necessary).
Proof: Let $x \in X$. Then $M_x \otimes N_x \cong \mathcal{O}_{X,x}$. Now tensor with the residue field of $\mathcal{O}_{X,x}$ and use linear algebra to conclude that $M_x / \mathfrak{m}_x M_x$ is $1$-dimensional. Since $M_x$ is of finite type over $\mathcal{O}_{X,x}$, Nakayama shows that $M_x$ is generated by just one element. Since $M$ is of finite type in a neighborhood of $x$, it follows that the generator at $x$ is also a generator in a neighborhood of $x$. Also $N$ has one generator, and their tensor product is a generator of $M \otimes N \cong \mathcal{O}_X$, which must be free. Thus also the generators of $M$ and $N$ are free.
But I don't know what happens in the general case. Here are some intermediate questions:
- Does it follow that $M$ is flat?
- Is the resulting morphism $M \to Hom(N,\mathcal{O}_X)$ an isomorphism?
- Is the claim true for $X$ a point, i.e. a local ring?
- Is the claim true if $X$ is an affine scheme and $M,N$ are quasi-coherent? (Thus in the question, replace $\mathcal{O}_X$ by a usual ring.)