Let $D$ be a division algebra over a number field $K$, and consider abelian varieties $A$ over the complex numbers, of dimension $g$ with an action of (an order of) $D$.  Is it known when this set is non-empty, and if so what is the dimension of this space?

I always thought this was a solved question but glancing over the literature it seems like it might be tricky. Is there a good reference?