Justification of the term "invertible sheaf" Let $X$ be a locally ringed space (or a scheme) and $M,N$ two $\mathcal{O}_X$-modules such that $M \otimes N \cong \mathcal{O}_X$. Does it follow that $M$ is invertible in the usual sense, namely that $M$ is locally free of rank $1$?
It is true if $M$ is locally of finite type (which is, of course, also necessary). 
Proof: Let $x \in X$. Then $M_x \otimes N_x \cong \mathcal{O}_{X,x}$. Now tensor with the residue field of $\mathcal{O}_{X,x}$ and use linear algebra to conclude that $M_x / \mathfrak{m}_x M_x$ is $1$-dimensional. Since $M_x$ is of finite type over $\mathcal{O}_{X,x}$, Nakayama shows that $M_x$ is generated by just one element. Since $M$ is of finite type in a neighborhood of $x$, it follows that the generator at $x$ is also a generator in a neighborhood of $x$. Also $N$ has one generator, and their tensor product is a generator of $M \otimes N \cong \mathcal{O}_X$, which must be free. Thus also the generators of $M$ and $N$ are free.
But I don't know what happens in the general case. Here are some intermediate questions:


*

*Does it follow that $M$ is flat?

*Is the resulting morphism $M \to Hom(N,\mathcal{O}_X)$ an isomorphism?

*Is the claim true for $X$ a point, i.e. a local ring?

*Is the claim true if $X$ is an affine scheme and $M,N$ are quasi-coherent? (Thus in the question, replace $\mathcal{O}_X$ by a usual ring.)

 A: Any $M'\to M$ that induces an epi after tensoring with $N$ must also induce an epi after tensoring with $N\otimes M$ and therefore must be an epi. And locally such a finitely generated $M'$ must exist.
A: Yes, for quasi-coherent sheaves on a scheme it is true that if $M \otimes N \cong \mathcal{O}_X$, then 
$M$ is locally free of rank one. It is enough to prove the 
REDUCTION  Let $M,N$ be $A$- modules such that $M \otimes_A N\cong A$. Then $M$ is projective of finite type.
Proof : We are given an isomorphism $f:M \otimes_A N\cong A$ . Say $\; f( \Sigma m_i\otimes n_i)= 1$ (FINITE index set!). The composition of the isomorphisms
$g_M:M \to M\otimes (N\otimes M):m\mapsto \Sigma m\otimes (n_i\otimes m_i)$
$assoc: M\otimes (N\otimes M) \to (M\otimes N) \otimes M:
m\otimes (n \otimes m') \mapsto (m\otimes n)\otimes m'$ 
$f_M:(M\otimes N) \otimes M \to M: (m\otimes n) \otimes m_1\mapsto f (m \otimes n).m_1 $
is the isomorphism $j:M\to M: m\mapsto \Sigma  f(m\otimes n_i).m_i$
By introducing the linear forms $\nu_i:M\to A: m\mapsto f(m\otimes n_i)$ we see that we have an isomorphism
$j:M\to M: m\mapsto \Sigma \nu _i(m).m_i$ and we deduce that for all $m\in M$ we can write  $m=\Sigma \nu _i(m).j^{-1}(m_i)$.
It is well known that the existence of such a dual basis $(j^{-1 }m_i, \nu_i)$ proves that $M$ is a finitely generated projective module. 
A: It seems to me that if $X$ is not factorial than one must be careful.
For instance, take the weighted projective space 
$X=\mathbb{P}(1,1,2)$, and consider the sheaf $\mathcal{O}_X(1)$, whose construction can be found for instance in Dolgachev's paper "Weighted projective varieties". 
Abstractely, $X$ is isomorphic to a quadric cone in $\mathbb{P}^3$, and the sheaf $\mathcal{O}_X(1)$ has only two sections, so it is not locally free (in fact, it corresponds to a single line in the cone). However, $\mathcal{O}_X(2)$ is locally free, since it corresponds to a hyperplane section (in fact, $X$ is 2-factorial); let $\mathcal{O}_X(-2):=\mathcal{O}_X(2)^*$.
Finally, take 
$M:=\mathcal{O}_X(-2) \otimes \mathcal{O}_X(1), \quad N:=\mathcal{O}(1)$.
Then $M \otimes N \cong \mathcal{O}_X$, but $M$ and $N$ are not locally free.
ADDED. As pointed out by D. Arapura in the comment below, there is actually a mistake in this example, since it is $not$ true that 
$\mathcal{O}_X(1) \otimes \mathcal{O}_X(1) \cong \mathcal{O}_X(2)$; in fact, these sheaves only agree on a dense open set. What is true is that
$\mathcal{O}_X(2a) \otimes \mathcal{O}_X(b) \to \mathcal{O}_X(2a+b)$
is an isomorphism for all integers $a$, $b$, but of course this cannot give any counterexample!
However, it is $always$ true that
$\mathcal{O}_X(a) \to Hom (\mathcal{O}_X(b), \mathcal{O}_X(a+b))$
is an isomorphism, see the paper of DELORME "Espaces projectifs anisotropes", p. 210.
In particular,
$\mathcal{O}_X(1) \to Hom (\mathcal{O}_X(-1), \mathcal{O}_X)$
is an isomorphism, even if $\mathcal{O}_X(1) \otimes \mathcal{O}_X(-1) \neq \mathcal{O}_X$.
