In fact many texts give equivalent statements to RH in terms of $M(N)$ and related arithmetic functions. In particular, RH is equivalent to the following: for any $\epsilon>0$, there is $N_0$ depending on $\epsilon$, such that for any $N>N_0$ we have $|M(N)|<N^{1/2+\epsilon}$.
Informally, this means that if RH fails, then $|M(N)|$ is occasionally much larger than $\sqrt{N}$, and vice versa. It is a subtle question how large $|M(N)|$ can really get if RH holds. For the state-of-the-art in this direction and for what more can be expected, see Soundararajan's paper.