The answer is yes under very mild assumptions. In fact there is the following result.
Proposition. Let $L$ be a $k$-ample line bundle on a normal, irreducible, projective variety $X$ with at most Cohen-Macauley singularities. Assume that the dimension of the locus of non-rational singularities of $X$ is at most $0$ and that $D \in |L|$ is a divisor such that $X-D$ is a local complete intersection. Then, under the restriction mapping, we have $\textrm{Pic}(X) \cong \textrm{Pic}(D)$ if $\dim X \geq 4+k$ and $\textrm{Pic}(X) \to \textrm{Pic}(D)$ is injective with torsion free cokernel if $\dim X = 3+k$.
The case you are interested in corresponds to $k=0$. For instance, if $X$ is smooth (of dimension at least $4$) then all the assumptions are fulfilled and what you want is true.
The Proposition above is a consequence of Hamm's Lefschetz Theorem. For further details, see Beltrametti-Sommese, The adjunction theory of complex projective varieties, Corollary 2.3.4 page 51.
Remark. Given an integer $k \geq 0$, a line bundle $L$ on a projective variety $X$ is said to be $k$-ample if $a L$ is spanned for some $a > 0$, and the morphism $X \to \mathbb{P}^{h^0(aL)-1}$ defined by $|a L|$ has all fibers of dimension $\leq k$. For $k=0$, this is one of the basic characterization of ampleness.