If $E$ is not CM, then the action of complex conjugation on $E$ depends on how it is embedded into $\mathbb{C}$. In particular, it could have different real subfields depending on which embedding you are using. When $E$ is CM, so that it is a totally imaginary quadratic extension of the totally real subfield $F$, then it has a unique complex conjugation that commutes with all automorphisms of $E$, such that $F$ contains precisely the elements that are fixed by complex conjugation in all embeddings. This lets you talk about unitarity and hermiticity for the abstract field $E$, and not just for some particular embedding.
Or, to put it another way, there can be more than one way to consistently define a complex conjugation on your field, unless it is CM, in which case there is a unique way to define it.