The following seems (I have to think about it) to work [EDIT: no, it doesn't]: By the universal property of the Albanese, we just have to construct a morphism $X \to \mathrm{Alb}(X)^\vee$:

Given a line bundle $\mathcal{L}$ on $X$, we can pull it back to $\mathcal{L}$ on $X \times \mathrm{Alb}(X)$, and we have to check that it is trivial when pulled back by the zero section and in $\mathrm{Pic}^0(\mathrm{Alb}(X))$ when pulled back by all $x \in X$ [EDIT: This is wrong.]. Then Mumford, Abelian Varieties, p. 125 gives us a unique $\phi: X \to \mathrm{Alb}(X)^\vee$ with $\mathcal{L} = (\phi \times 1_{\mathrm{Alb}(X)})^*\mathcal{P}$.

For the induced map $\psi: \mathrm{Alb}(X) \to \mathrm{Alb}(X)^\vee$ to be a polarisation, we have to check that $\psi^\vee = \psi$ in $\mathrm{Hom}(\mathrm{Alb}(X), \mathrm{Alb}(X)^\vee)$.

Conversely, a polarisation is given by a line bundle on $\mathrm{Alb}(X)$, which we can pull back by the Abel-Jacobi map.