The following answer expands on my comment.

We use Fulton's definition of the intersection product. Consider the diagonal embedding $\Delta$ of $A$ in $A \times A$ (which is regular) and intersect this with $X \times Y$. Since $A$ is an abelian variety, the normal bundle $E$ of $\Delta$ is trivial. The normal cone $C$ of $X \times Y \cap \Delta$ is an effective cycle in $E$ and the intersection product is the pullback of this cycle to $\Delta$ via the zero section. Connected components of $C$ map to connected components of $X \cap Y$ and the claim is that each such component gives a non-zero contribution. This follows from the following claim:

Let $Z$ be any variety and $E$ a trivial vector bundle on $Z$. Then the Gysin map (induced by the zero section) from $CH^*(E)$ to $CH^*(Z)$ preserves effective cycles.

This can be seen using Proposition 3.3 in Fulton's book where this Gysin map is expressed as a product followed by a pushforward. In the case $E$ is trivial, the product involved can be seen to be effective since it is given by repeated intersection with divisors in a base point free linear system. Pushforward always preserves effectivity, so the claim follows.