When $A$ is a CM abelian variety of dimension $1$ (i.e., an elliptic curve), then we have a result that if it has CM by a maximal order then it has a model over a number field $F$ where $F$ is the Hilbert class field of the CM field if $A$ has CM by the maximal order and is otherwise a ray class field. In particular, if $A$ has CM by a non-maximal order then it has a "strictly larger" field of definition. (Experts, please excuse some sloppiness here.)

Is the analogous result true in general? I.e., is it true that if we take two CM abelian varieties $A$ and $A^\prime$ with CM by $(K,\Phi)$ but such that $A$ has CM by the maximal order $\mathcal{O}_K\subseteq K$ and $A^\prime$ has CM by a non-maximal order $\mathcal{O}\subseteq K$ that for the analogous model of $A^\prime$ over $F^\prime$ and $A$ over $F$ we must have that $[F^\prime:F]>1$?