As dhy said, this follows if we know that the fibers $M(r,\xi)$ are rational. In fact, it suffices to know that they are unirational - we can deduce that their image in any map to an abelian variety is a point, hence any map from $M(r,d)$ to an abelian variety factors through the determinant map, and thus the Alabanese of $M(r,d)$ is $\operatorname{Jac}^d$.

Their unirationality is not so hard to prove.

Take $L$ ample enough that $V \otimes L$ is globally generated for every stable vector bundle of rank $r$ and determinant $\xi$. (It suffices to have $H^1( X, V \otimes L (-P)) =0$ for all points $P$, i.e. it suffices to have $H^0(X, V^\vee \otimes K_X \otimes L^{\vee} (P))$, so it suffices to have $\deg L > 2g-1 + \frac{ \deg xi}{r}$.)

Then among maps $(L^{-1})^{n-1} \to V$, those which have rank $< n-1$ at a point $P$ form a codimension $2$ subset, so those which have rank $<n-1$ at any point form a codimension $1$ subset, and thus there exists a map $(L^{-1})^{n-1} \to V$ with full rank at every point. Hence the quotient is a line bundle, which because $\det V = \xi$, must be $ \xi \otimes L^{\otimes (n-1)}$. So $V$ is an extension of $\xi \otimes L^{\otimes (n-1)}$ by $(L^{-1})^{n-1}$.

Now there is an open subset of $\operatorname{Ext}^1 ( \xi \otimes L^{\otimes (n-1)}, (L^{-1})^{n-1} )$ parameterizing stable vector bundles, which maps to $M(r,\xi)$. By the above argument this map is surjective, so $M(r,\xi)$ is unirational.

As dhy said, this follows if we know that the fibers $M(r,\xi)$ are rational. In fact, it suffices to know that they are unirational - we can deduce that their image in any map to an abelian variety is a point, hence any map from $M(r,d)$ to an abelian variety factors through the determinant map, and thus the Albanese of $M(r,d)$ is $\operatorname{Jac}^d$.

Their unirationality is not so hard to prove.

Take $L$ ample enough that $V \otimes L$ is globally generated for every stable vector bundle of rank $r$ and determinant $\xi$. (It suffices to have $H^1( X, V \otimes L (-P)) =0$ for all points $P$, i.e. it suffices to have $H^0(X, V^\vee \otimes K_X \otimes L^{\vee} (P))$, so it suffices to have $\deg L > 2g-1 + \frac{ \deg xi}{r}$.)

Then among maps $(L^{-1})^{n-1} \to V$, those which have rank $< n-1$ at a point $P$ form a codimension $2$ subset, so those which have rank $<n-1$ at any point form a codimension $1$ subset, and thus there exists a map $(L^{-1})^{n-1} \to V$ with full rank at every point. Hence the quotient is a line bundle, which because $\det V = \xi$, must be $ \xi \otimes L^{\otimes (n-1)}$. So $V$ is an extension of $\xi \otimes L^{\otimes (n-1)}$ by $(L^{-1})^{n-1}$.

Now there is an open subset of $\operatorname{Ext}^1 ( \xi \otimes L^{\otimes (n-1)}, (L^{-1})^{n-1} )$ parameterizing stable vector bundles, which maps to $M(r,\xi)$. By the above argument this map is surjective, so $M(r,\xi)$ is unirational.