Let $X$ be a smooth projective variety, and $D\subset X$ a smooth nef and big divisor. Assume that the restriction map $Pic(X)\rightarrow Pic(D)$ is an isomorphism over $\mathbb{Q}$.
Under which hypothesis may we conclude that there exists an isomorphism $\mathrm{Pic}(X)\rightarrow \mathrm{Pic}(D)$ over $\mathbb{Z}$ ?
I am particularly interested in the case $dim(X)=3$.