If $E$ is not CM, then the action of complex conjugation on $E$ depends on how it is embedded into $\mathbb{C}$.  In particular, it could have different real subfields depending on which embedding you are using.  When $E$ is CM, so that it is a totally imaginary quadratic extension of the totally real subfield $F$, then it has a unique complex conjugation that commutes with all automorphisms of $E$, such that $F$ contains precisely the elements that are fixed by complex conjugation in all embeddings.  This lets you talk about unitarity and hermiticity for the abstract field $E$, and not just for some particular embedding, which I would imagine could be problematic if not impossible to say anything useful about.