I missed the irreducibility hypothesis above, but there are similar examples where $Y$ is irreducible.  The following example is <b>not</b> complete, because I have not computed the primitive cohomology in the middle dimension.  Also, this example uses twisted versions of $\mathbb{CP}^1 \times \mathbb{CP}^1$, but it might be better to use twisted versions of $(\mathbb{CP}^1)^r = \mathbb{CP}^1 \times \dots \times \mathbb{CP}^1$ with $r>2$.  

Let $B$ be a connected, complex projective manifold.  Let $$f:C\to B$$ be a finite, &eacute;tale morphism of degree $2$ with $C$ connected.  Denote the associated $f$-involution of $C$ by $$i:C\to C.$$  Denote by $P$ the manifold $C\times \mathbb{CP}^1$, and denote by $$\rho:P\to C,$$ the projection onto the first factor.  Denote by $$\pi:X\to B$$ the Weil restriction of scalars for $\rho$ relative to $f$, i.e., what Grothendieck refers to as $\Pi_{P/C/B}$ in his Bourbaki seminars in the Hilbert functor.  Very explicitly, $X$ parameterizes length $2$, finite, closed subschemes $Z$ of $P$ such that the induced morphism $$f\circ \rho:Z\to C$$ is constant and such that the induced morphism $$\rho:Z\to C$$ is a closed immersion.  There is an associated finite, &eacute;tale morphism of degree $2$, $$ h:C\times \mathbb{CP}^1 \times \mathbb{CP}^1 \to X, \ \ (c,t,u) \mapsto Z=\{(c,t),(i(c),u)\}.$$  The associated $h$-involution is $$\iota:C\times \mathbb{CP}^1 \times \mathbb{CP}^1 \to C\times \mathbb{CP}^1 \times \mathbb{CP}^1, \ \ (c,t,u) \mapsto (i(c),u,t).$$

For every sufficiently ample invertible sheaf $\mathcal{L}$ on $B$, the associated invertible sheaf $\text{pr}_1^*\mathcal{L}\otimes \text{pr}_{\mathbb{CP}^1}^*\mathcal{O}(1)$ on $B\times \mathbb{CP}^1$ is also ample and globally generated.  The Cartier divisor $E$ of a general section will be $\text{pr}_1$-flat away from a codimension $2$ closed subset $B_E$ of $B$.  Define $D$ to be the inverse image of $E$ under $f\times \text{Id}_{\mathbb{CP}^1}$ as a Cartier divisor in $C\times \mathbb{CP}^1$.  The product $D\times \mathbb{CP}^1$ in $P\times \mathbb{CP}^1 = C \times \mathbb{CP}^1 \times \mathbb{CP}^1$ is nef but not ample.  For the same reason, the pullback divisor $\iota^*(D\times \mathbb{CP}^1)$ is nef but not ample.  However, the sum of these two divisors is ample, and it is $\widetilde{j}$-invariant.  Thus it is the $h$-pullback of an ample divisor $Y$ in $X$, $$h^*Y = (D\times \mathbb{CP}^1) + \iota^*(D\times \mathbb{CP}^1).$$  Of course $Y$ is the image under $h$ of $D\times \mathbb{CP}^1$, so that $Y$ is irreducible.  The open complement of $Y$ maps surjectively to $B\setminus B_E$, and it is a (topologically) locally trivial fiber bundle with fiber $\mathbb{C}^1\times \mathbb{C}^1$.  Since the cohomology of the fiber is trivial, by the Leray spectral sequence, the following pullback map on cohomology is an isomorphism, $$\pi^*:H^*(B\setminus B_E;\mathbb{Q}) \to H^*(X\setminus Y;\mathbb{Q}).$$  Restricted over $B\setminus B_E$, the divisor $E$ is the graph of a morphism of varieties, $$s:(B\setminus B_E)\to \mathbb{CP}^1.$$  Now for the usual antipodal map on $\mathbb{CP}^1$, $$a:\mathbb{CP}^1\to \mathbb{CP}^1, \ \ a(z) = -1/\overline{z},\ \text{i.e., } [z_0,z_1] \mapsto [\overline{z}_1,-\overline{z}_0],$$ the composition $a\circ s(B\setminus B_E)$ is everywhere disjoint from $E$.  Thus, pullback by $a\circ s$ defines an inverse isomorphism of $\pi^*$, $$(a\circ s)^*:H^*(X\setminus Y;\mathbb{Q}) \to H^*(B\setminus B_E;\mathbb{Q}).$$  

We can easily arrange that $B_E$ is a complete intersection of two ample divisors in $B$.  Then $B\setminus B_E$ is covered by two open affine subvarieties, so that the affine covering number equals $2-1=1$.  Thus, $H^d(B\setminus B_E;\mathbb{Q})$ is zero for $d\geq 2 + \text{dim}_{\mathbb{C}}(B)$.  But $n=2+\text{dim}_{\mathbb{C}}(B)$ is the (complex) dimension of $X$.  So the primitive cohomology of $X$ in this degree is the kernel of $$c_1(Y)\cap - : H^n(X;\mathbb{Q}) \to H^{n+2}(X;\mathbb{Q}).$$  Thus, if the primitive cohomology of $X$ in the middle degree is nonzero, then the pullback map $j^*$ is not injective.