I think this is a counterexample. Let $V = \mathbf P^2 \times \mathbf P^1$, $L= \mathscr{O}_{P^2} \sqtimes \mathscr{O}_{P^1}(1)$. Let $\ell_1, \ell_2$ be two distinct lines in $\mathbf{P}^2$ and let $X_i=\ell_i \times \mathbf P^2$. Then $Z=X_1 \cap X_2 = \{\mathrm{pt}\}\times \mathbf P^1$, $L|_Z$ has degree $1$ but $L|_{X_i}$ has degree $0$

I think this is a counterexample. Let $V = \mathbf P^2 \times \mathbf P^1$, $L= \mathscr{O}_{P^2} \boxtimes \mathscr{O}_{P^1}(1)$. Let $\ell_1, \ell_2$ be two distinct lines in $\mathbf{P}^2$ and let $X_i=\ell_i \times \mathbf P^2$. Then $Z=X_1 \cap X_2 = \{\mathrm{pt}\}\times \mathbf P^1$, $L|_Z$ has degree $1$ but $L|_{X_i}$ has degree $0$