When defining unitary groups over number fields, one usually takes $F$ to be a totally real number field, $E$ a CM quadratic extension of $F$, and $V$ a hermitian space attached to $E/F$. Then $U(V)$ is simply the isometry group for the hermitian form attached to $V$.

My question is: why does one take $F$ to be totally real and $E$ CM? The definition makes perfect sense without this, but most references (at least in the context of automorphic forms) make these assumptions.

Hermitian type, i.e. it acts on a Hermitian symmetric domain $X$. If you choose a congruence subgroup $\Gamma\subset G(\mathbb{Q})$, then $X/\Gamma$ is a Shimura variety. $\endgroup$