As Tony Pantev points out, it is easy to make an example by taking non-algebraic classes.
If you impose that $t$ is algebraic, and take $X=Y$, you are very close to stating Grothendieck's Conjecture D. Let $X$ be smooth and projective. The conjecture is that any algebraic class in $H^*(X)$, which is orthogonal to all algebraic classes, is $0$zero torsion.