Certainly it does not suffice to take the schematic closure if $S$ is nonreduced. For instance, let $S$ be $\text{Spec}\ k[x,y]/\langle x^2, xy \rangle$. Let $A$ be $E \times_{\text{Spec} k} S$, where $E$ is an elliptic curve over $k$ with specified zero point $z$. Let $p \in S$ be the closed point with maximal ideal $\langle x,y\rangle$. Let $U$ be the open complement in $A$ of the closed point $(z,p)$. Let $\zeta:S\to A$ be the zero section with image $\{z\}\times S$. Let $D$ be the intersection of $U$ with the image Cartier divisor $\zeta(S)$. Then $D'$ is the underlying reduced scheme of $\zeta(S)$, and this is not a Cartier divisor in $A$, nor is it flat over $S$.