Let $X$ be a smooth complex projective variety and $p:Y\to X$ be a smooth $\mathbb CP^k$-bundle (i.e. locally trivial in analytic topology). Suppose that there exists a line bundle $L$ on $Y$ that restricts to $\mathcal O(1)$ on each $\mathbb CP^k$-fibre.

Question. Is it true that there is a line bundle $L'$ on $X$ such that $p^*L'\otimes L$ is very ample on $Y$?

I think one should be able to prove this using Serre's vanishing, but I can't prove it so far.