On the jacobian origin of CM abelian varieties

Let $K$ be a CM field of degree $2n$ over $\mathbf{Q}$ and let $\mathcal{O}_K$ be its ring of integers. Let $\Phi=(\phi_1,\ldots,\phi_n)$ be a CM type of K. Then it is known that complex torus $A:=\mathbf{C}^n/\Phi(\mathcal{O}_K)$ is algebraic. For instance one may construct a Riemann form on $A$ in the following way: choose $\xi\in K$ such that $\xi^2\in K^+$ is totally negative and $\Im(\phi_i(\xi))>0$ for all $i$. Then for a suitable integer $m$ one has that the alternating form $$E(z,w):=m\sum_i \phi_i(\xi)(\overline{z}_iw_i-z_i\overline{w}_i)$$ is a Riemann form of $A$.

Q: Having an abelian varietey $A$ as above is it reasonable to expect it to be the isogeneous to the Jacobian of a curve?

Of course if $n=1$ then the answer is always true but what about $n>1$? Are there some obvious obstructions that prevent $A$ to have a Jacobian origin?

-
When $n < 4$, $A$ is isogenous to the Jacobian of a stable curve because the Torelli locus is dense. This should imply that $A$ is isogenous to the Jacobian of a smooth curve because $A$ is simple (at least if the CM-type is primitive).
If $n \geq 4$, then conditional results of Chai-Oort (see their recent Annals paper) imply that $A$ is not necessarily isogenous to the Jacobian of a curve. Tsimerman then proved this unconditionally. I would guess that the set of CM points whose isogeny orbit is disjoint from the Torelli locus is dense in $\mathcal{A_g}$, but I don't know.
Ari, is $\mathcal{A}_g$ the moduli space of principally polarized abelian varieties? Even if this Torelli locus is dense (Zariski or analytically) how does it say anything about the Jacobian origin of a given principally polarized CM abelian variety? –  Hugo Chapdelaine Aug 18 '12 at 15:04
Sorry, I was a bit terse. The open Torelli locus (i.e. Jacobians of smooth curves) is dense in $\mathcal{A}_n$ (the moduli space of $n$-dimensional ppav) for $n < 4$, so the closure is the whole thing. But it's known that anything in (closure minus open locus) is a product of Jacobians (see Mumford's Woods Hole notes). So if $A$ is simple then it's isogenous to a Jacobian. –  Ari Shnidman Aug 18 '12 at 16:53
So @Ari, do you know what kind of obstruction when $g\geq 4$ prevents an abelian variety to be the Jacobian of a curve? –  Hugo Chapdelaine Aug 19 '12 at 17:40