I missed the irreducibility hypothesis above, but there are similar examples where $Y$ is irreducible. The following example works assuming that the cup product with $[Y]$ defined below preserves the graded subring $H^*(B;\mathbb{Q})[x_1,\dots,x_r]/\langle x_1^2,\dots,x_r^2\rangle.$ I believe that this is true, and I will try to get back to this when I have more time.
Let $C$ be a connected, complex projective manifold, and let $i$ be a biholomorphism, $$i:C\to C,$$ that defines a free action of the cyclic group of order $r\geq 2.$ Denote the quotient of this free action by $$f:C\to B, \ \ f(i(c)) = f(c).$$ Denote by $P$ the manifold $C\times \mathbb{CP}^1$, and denote by $$\rho:P\to C,$$ the projection onto the first factor. Denote by $$\pi:X\to B$$ the Weil restriction of scalars for $\rho$ relative to $f$, i.e., what Grothendieck refers to as $\Pi_{P/C/B}$ in his Bourbaki seminars in the Hilbert functor. Very explicitly, $X$ parameterizes length $r$, finite, closed subschemes $Z$ of $P$ such that the induced morphism $$f\circ \rho:Z\to C$$ is constant and such that the induced morphism $$\rho:Z\to C$$ is a closed immersion. There is an associated finite, étale morphism of degree $r$, $$ h:C\times (\mathbb{CP}^1)^r \to X,$$ $$(c,t_0,\dots,t_{r-1}) \mapsto \{(c,t_0),(i(c),t_1), (i(i(c)),t_2),\dots, (i^{r-1}(c),t_{r-1})\}.$$ This the quotient by the free action of the degree $r$ cyclic group generated by the following biholomorphism, $$\iota:C\times (\mathbb{CP}^1)^r \to C\times (\mathbb{CP}^1)^r, \ \ (c,t_0,\dots,t_{r-1},t_{r-1}) \mapsto (i(c),t_1,\dots,t_r,t_0).$$
For every sufficiently ample invertible sheaf $\mathcal{L}$ on $B$, the associated invertible sheaf $\text{pr}_1^*\mathcal{L}\otimes \text{pr}_{\mathbb{CP}^1}^*\mathcal{O}(1)$ on $B\times \mathbb{CP}^1$ is also ample and globally generated. The Cartier divisor $E$ of a general section will be $\text{pr}_1$-flat away from a codimension $2$ closed subset $B_E$ of $B$. Define $D$ to be the inverse image of $E$ under $f\times \text{Id}_{\mathbb{CP}^1}$ as a Cartier divisor in $C\times \mathbb{CP}^1$. The following Cartier divisor in $C\times(\mathbb{CP}^1)^r$ is ample and $\iota$-invariant, hence equal to the inverse image under $h$ of an ample Cartier divisor $Y$ in $X$, $$h^*Y = \sum_{i=1}^r \text{pr}_{1,i+1}^*D.$$
Of course $Y$ is the image under $h$ of $D\times (\mathbb{CP}^1)^{r-1}$, so that $Y$ is irreducible. The open complement of $Y$ maps surjectively to $B\setminus B_E$, and it is a (topologically) locally trivial fiber bundle with fiber $(\mathbb{C}^1)^r= \mathbb{C}^r.$ Since the cohomology of the fiber is trivial, the Leray spectral sequence degenerates, and the following pullback map on cohomology is an isomorphism, $$\pi^*:H^*(B\setminus B_E;\mathbb{Q}) \to H^*(X\setminus Y;\mathbb{Q}).$$
We can easily arrange that $B_E$ is a complete intersection of two ample divisors in $B$. Then $B\setminus B_E$ is covered by two open affine subvarieties, so that the affine covering number equals $2-1=1$. Thus, $H^d(B\setminus B_E;\mathbb{Q})$ is zero for $d\geq 2 + \text{dim}_{\mathbb{C}}(B)$. Note that the complex dimension of $X$ equals $$n=\text{dim}_{\mathbb{C}}(X) = r + m, \ \ m:=\text{dim}_{\mathbb{C}}(B).$$ Thus, if the primitive cohomology $H^{n-k}_{\text{prim}}(X;\mathbb{Q})$ is nonzero for any integer $k$ with $0\leq k \leq r-2$, then necessarily $j^*$ in degree $n-k$ is not injective on primitive cohomology.
In particular, it appears that cup product against $Y$ preserves the following subring of the cohomology ring of $X$, $$H^*(B;\mathbb{Q})[x_1,\dots,x_r]/\langle x_1^2,\dots,x_r^2\rangle^{\mathfrak{S}_r}, \ \ h^*x_i \text{pr}_{i+1}^*c_1(\mathcal{O}(1)) \in H^*(C\times (\mathbb{CP}^1)^r;\mathbb{Q}).$$ As a module over the ring $H^*(B;\mathbb{Q})$, this is generated by the elementary symmetric polynomials in the classes $x_i$, $$\sigma_0 = 1 , \sigma_1 = \sum_i x_i, \sigma_2 = \sum_{i_1<i_2}x_{i_1}x_{i_2}, \dots, \sigma_r = x_1\cdots x_r, \ \ \text{deg}(\sigma_i) = 2i.$$
Now assume further that the complex dimension $m=\text{dim}_{\mathbb{C}}(B)$ is at least as large as $r+2$. In the middle degree $m + r$, the $\mathbb{Q}$-dimension of the associated graded piece of this module equals the sum of the Betti numbers of $B$, $$\text{dim}_{\mathbb{Q}}H^*(B;\mathbb{Q})[x_1,\dots,x_r]/\langle x_1^2,\dots,x_r^2\rangle^{\mathfrak{S}_r} = \beta_{m-r}(X) + \beta_{m+2-r}(X) + \dots + \beta_{m-2+r}(X) + \beta_{m+r}(X).$$ The associated graded piece in degree $m+r+2$ equals $$\beta_{m-2-r}(X) + \beta_{m-r}(X) + \dots + \beta_{m-4+r}(X) + \beta_{m-2+r}(X).$$ Thus, the discrepancy between the first dimension and the second dimension equals $$\beta_{m+r}(X) - \beta_{m-r-2}(X).$$ By Poincaré duality, of course this is the same as $\beta_{m-r}(X) - \beta_{m-r-2}(X)$. Thus, if this discrepancy is strictly positive, then there is a nonzero primitive cohomology in the middle degree on $X$ whose image under $j^*$ equals zero. This will be the case, for instance, if $m$ equals $r+2$, so that $\beta_{m-r-2}(X)$ equals $\beta_0(X)=1$, and if $\beta_2(X)\geq 2$.