I missed the irreducibility hypothesis above, but there are similar examples where $Y$ is irreducible.  The following example is <b>not</b> complete, because I have not computed the primitive cohomology in the middle dimension.  

Let $C$ be a connected, complex projective manifold, and let $i$ be a biholomorphism, $$i:C\to C,$$ that defines a free action of the cyclic group of order $r\geq 2.$  Denote the quotient of this free action by $$f:C\to B, \ \ f(i(c)) = f(c).$$ Denote by $P$ the manifold $C\times \mathbb{CP}^1$, and denote by $$\rho:P\to C,$$ the projection onto the first factor.  Denote by $$\pi:X\to B$$ the Weil restriction of scalars for $\rho$ relative to $f$, i.e., what Grothendieck refers to as $\Pi_{P/C/B}$ in his Bourbaki seminars in the Hilbert functor.  Very explicitly, $X$ parameterizes length $r$, finite, closed subschemes $Z$ of $P$ such that the induced morphism $$f\circ \rho:Z\to C$$ is constant and such that the induced morphism $$\rho:Z\to C$$ is a closed immersion.  There is an associated finite, &eacute;tale morphism of degree $r$, $$ h:C\times (\mathbb{CP}^1)^r \to X,$$ $$(c,t_0,\dots,t_{r-1}) \mapsto \{(c,t_0),(i(c),t_1), (i(i(c)),t_2),\dots, (i^{r-1}(c),t_{r-1})\}.$$  This the quotient by the free action of the degree $r$ cyclic group generated by the following biholomorphism, $$\iota:C\times (\mathbb{CP}^1)^r \to C\times (\mathbb{CP}^1)^r, \ \ (c,t_0,\dots,t_{r-1},t_{r-1}) \mapsto (i(c),t_1,\dots,t_r,t_0).$$

For every sufficiently ample invertible sheaf $\mathcal{L}$ on $B$, the associated invertible sheaf $\text{pr}_1^*\mathcal{L}\otimes \text{pr}_{\mathbb{CP}^1}^*\mathcal{O}(1)$ on $B\times \mathbb{CP}^1$ is also ample and globally generated.  The Cartier divisor $E$ of a general section will be $\text{pr}_1$-flat away from a codimension $2$ closed subset $B_E$ of $B$.  Define $D$ to be the inverse image of $E$ under $f\times \text{Id}_{\mathbb{CP}^1}$ as a Cartier divisor in $C\times \mathbb{CP}^1$.  The following Cartier divisor in $C\times(\mathbb{CP}^1)^r$ is ample and $\iota$-invariant, hence equal to the inverse image under $h$ of an ample Cartier divisor $Y$ in $X$,  $$h^*Y = \sum_{i=1}^r \text{pr}_{1,i+1}^*D.$$
Of course $Y$ is the image under $h$ of $D\times (\mathbb{CP}^1)^{r-1}$, so that $Y$ is irreducible.  The open complement of $Y$ maps surjectively to $B\setminus B_E$, and it is a (topologically) locally trivial fiber bundle with fiber $(\mathbb{C}^1)^r= \mathbb{C}^r.$  Since the cohomology of the fiber is trivial, the Leray spectral sequence degenerates, and the following pullback map on cohomology is an isomorphism, $$\pi^*:H^*(B\setminus B_E;\mathbb{Q}) \to H^*(X\setminus Y;\mathbb{Q}).$$  

We can easily arrange that $B_E$ is a complete intersection of two ample divisors in $B$.  Then $B\setminus B_E$ is covered by two open affine subvarieties, so that the affine covering number equals $2-1=1$.  Thus, $H^d(B\setminus B_E;\mathbb{Q})$ is zero for $d\geq 2 + \text{dim}_{\mathbb{C}}(B)$.  Note that the complex dimension of $X$ equals $$n=\text{dim}_{\mathbb{C}}(X) = r + \text{dim}_{\mathbb{C}}(B).$$  Thus, if the primitive cohomology $H^{n-k}_{\text{prim}}(X;\mathbb{Q})$ is nonzero for any integer $k$ with $0\leq k \leq r-2$, then necessarily $j^*$ in degree $n-k$ is not injective on primitive cohomology.

In particular, it appears that cup product against $Y$ preserves the following subring of the cohomology ring of $X$, $$H^*(B;\mathbb{Q})[x_1,\dots,x_r]/\langle x_1^2,\dots,x_r^2\rangle^{\mathfrak{S}_r}, \ \ h^*x_i \text{pr}_{i+1}^*c_1(\mathcal{O}(1)) \in H^*(C\times (\mathbb{CP}^1)^r;\mathbb{Q}).$$  As a module over the ring $H^*(B;\mathbb{Q})$, this is generated by the elementary symmetric polynomials in the classes $x_i$, $$\sigma_0 = 1 , \sigma_1 = \sum_i x_i, \sigma_2 = \sum_{i_1<i_2}x_{i_1}x_{i_2}, \dots, \sigma_r = x_1\cdots x_r, \ \ \text{deg}(\sigma_i) = 2i.$$