Let $A$ be an abelian variety over $\overline{\mathbb{Q}}$ and assume that $A$ has complex multiplication by the ring of integers of a CM field $K$. Then $A$ has potentially good reduction, that is: there exists a number field $F$ and an abelian scheme $\mathcal{A} \to \mathrm{Spec}(\mathcal{O}_F)$ whose generic fiber is isomorphic $A$ after extending the scalars from $F$ to $\overline{\mathbb{Q}}$.
The question is: how to determine $F$ in terms of $K$?
[EDIT: The following is false: I'm more familiar with the case of elliptic curve, in which case one can choose $F$ to be the Hilbert class field of $K$, that is, the maximal abelian unramified extension]
Does something similar hold for higher dimensional abelian varieties?