Does Albanese construction yield a morphism to moduli of abelian varieties?

Question

Let $M_h$ be the (coarse) moduli space of polarized manifolds with Hilbert function $h$. I would like to know if the albanese $Alb(X)$ of a polarized manifold $X$ gives rise to a morphism $M_h\to A_{g,\delta}$ to the moduli of abelian varieties with polarization type $\delta$? If not, what conditions should we add so that we get such a morphism?