Welcome new contributor. This is not true, even if $X$ is smooth. One example permutes the role of $X$ and $Y$ in my previous example.
Let $X$ be a smooth, geometrically connected, projective curve of genus $g>0$. Let $f:X\to Y$ be the normalization of a nodal curve with a single node $p$ that is a $k$-rational point. For instance, $Y$ could be a nodal plane quartic, and $X$ could be the normalization (a genus $3$ curve). Assume that the preimage of $\{p\}$ in $X$ is split, i.e., $\{r',r''\}$ for $k$-rational points $r',r''$ of $X$.
Let $V$ be the open complement of $\{r',r''\}$ in $X$. Denote the graph morphism of the restriction of $f$ to $V$ as follows, $$\Gamma:V\to V\times Y.$$ The image of this graph morphism is a prime Cartier divisor in $V\times Y$. Denote by $L$ the invertible sheaf on $V\times Y$ associated to this Cartier divisor.
The pullback of this Cartier divisor to $V\times X$ does extend to a Cartier divisor on $X\times X$. Every such extension is of the form $$D_{c',c''} = \underline{\Delta} + \text{pr}_1^*\left(c' \underline{r'} + c''\underline{r''}\right).$$
For each of these extended Cartier divisors, the restrictions over $X\times \{r'\}$ and over $X\times \{r''\}$ are not rationally equivalent. Indeed, if they were, then $\underline{r'}$ and $\underline{r''}$ would be rationally equivalent, so that the genus $g$ equals $0$. (This was my reason for working with smooth curves of positive genus.) Since $X\times X$ is smooth, the homomorphism from the group of rational equivalence classes of Cartier divisors to the Picard group is an isomorphism. Thus, every invertible sheaf on $X\times X$ that extends the pullback of $L$ has non-isomorphic restrictions over $X\times\{r'\}$ and over $X\times\{r''\}$. Therefore each extended invertible sheaf on $X\times X$ is not isomorphic to the pullback of an invertible sheaf on $X\times Y$.
Edit. In the example above, for every Zariski cover $Y'\to Y$, the same result holds. However, there is an étale cover $Y'\to Y$ such that the invertible sheaf extends to $X\times Y'$. For an example where there is no such extension even after an étale cover, instead of letting $X\to Y$ be the normalization of a nodal curve, let is be the normalization of a cuspidal curve. Then the same construction gives an invertible sheaf $L$ on $V\times Y$ such that for every étale cover $Y'\to Y$, there is no extension of the invertible sheaf to $X\times Y'$.