I am just posting my comment as an answer. Since $f:X\to S$ is smooth of finite type, in particular it is of finite presentation. By limit theorems, after replacing $S$ by the open in an open affine covering, there exists a Cartesian diagram, $$\begin{array}{ccc} X & \xrightarrow{f} & S \\ u~\downarrow & & \downarrow~v \\ X_0 & \xrightarrow{f_0} & S_0\end{array}, $$ such that $f_0$ is smooth and finite type, and such that $S_0$ is integral and Noetherian (even a finite type affine scheme over $\text{Spec}\ \mathbb{Z}$). Up to replacing $S_0$ by a finite type $S_0$-scheme through which $v$ factors, also there exists $\mathcal{E}_0$ on $X_0$ whose pullback by $u$ equals $\mathcal{E}$. Finally, up to replacing $S_0$ by the closure of the image of $v$, assume that $S_0$ is integral and that $v$ is dominant. For every $s$ in $S$ with image point $s_0$ in $S$, the fiber $X_s=\text{Spec}\ \kappa(s)\times_S X$ is the base change of the fiber $X_{s_0} = \text{Spec}\ \kappa(s_0)\times_{S_0} X_0$ via the field extension $\kappa(s_0)\hookrightarrow \kappa(s)$. If the pullback of $\mathcal{E}_0$ to $X_{s_0}$ is torsion-free, then also the flat base change by $\kappa(s_0)\hookrightarrow \kappa(s)$ is also torsion-free. Therefore it suffices to prove the result for $f_0$ and $\mathcal{E}_0$. Without loss of generality, assume that $S$ is Noetherian.

By the Generic Flatness Theorem, there exists a dense open subset of $S$ over which $\mathcal{E}$ is $S$-flat. Up to replacing $S$ by this dense open subset, assume that $\mathcal{E}$ is $S$-flat. By Théorème 12.1.1(iii) on p. 174 of the following,

MR0217086 (36 #178)
Grothendieck, A.
Éléments de géométrie algébrique. IV.
Étude locale des schémas et des morphismes de schémas. III.
Inst. Hautes Études Sci. Publ. Math. No. 28 1966 255 pp.
http://www.numdam.org/article/PMIHES_1966__28__5_0.pdf

the following set of points $x$ in $X$ is an open subset $V$ of $X$: those $x$ such that the restriction of $\mathcal{E}$ to the fiber $X_{f(s)}$ has no embedded primes containing the point $x$ of the fiber $X_{f(s)}$.

The complement $C=X\setminus V$ is a closed subset. By hypothesis, $C$ is disjoint from the fiber $X_\eta$ over the generic point $\eta$ of $S$. Thus, $f(C)$ is a constructible subset of $S$ that does not contain $\eta$. Therefore the closure of $f(C)$ is a closed subset of $S$ that does not contain $\eta$. For the open complement $U$ in $S$ of the closure of $f(C)$, the inverse image $f^{-1}(U)$ is contained in $V$. Therefore, for every $s\in U$, the restriction of $\mathcal{E}$ to the fiber $X_s$ has no embedded primes, i.e., the restriction is torsion-free.