Suppose $X$ is a smooth projective variety over a field $k\subset \mathbb{C}$. Let $CH^r(X)_{hom}$ be the Chow group of codimension $r$ cycles defined over $k$ and homologous to zero. The usual Abel-Jacobi map is defined on $CH^r(X(\mathbb{C}))_{hom}$ taking values in the Griffiths intermediate Jacobian $IJ(H^*(X))$.
Question: If this is an abelian variety, is it defined over the field $k$?