Over a field of characteristic zero, your result is true. This is Corollary III.10.7 in Hartshorne.  

<hr>

In characteristic $p$ no. The simplest example is to take $k$ an algebraically closed field and map $\mathbb{A}^1_k$ to itself by $x \mapsto x^p$. For every $t \in k$, the fiber above $t$ is $\mathrm{Spec}\ k[x]/(x^p-t) \cong \mathrm{Spec} \ k[y]/y^p$ where the isomorphism is $x-t^{1/p} = y$. So every fiber is singular.

There is a more interesting counter-example due to Serre: Let $k$ be an algebraically closed field of characteristic $2$. Consider the planar cubic
$$ F_{a,b,c}(x,y,z) :=a  (y^2 z + y z^2) + b (x^2 z + x z^2) + c  (x^2 y + x y^2) \quad (*)$$
We leave it to the reader to check that $F=0$ is singular at $(\sqrt{a}:\sqrt{b}:\sqrt{c})$. Generically, the singularity is a <strike>node</strike> cusp. Choose $(a_1, b_1, c_1)$ and $(a_2, b_2, c_2)$ in $k^3$ and try to map $\mathbb{P}^2_k \to \mathbb{P}^1_k$ by
$$(x:y:z) \mapsto (F_{(a_1,\ b_1,\ c_1)}(x,y,z) : F_{(a_2,\ b_2,\ c_2)}(x,y,z))$$
Then the fiber over $(t_1:t_2)$ is $F_{(t_1 a_1+t_2 a_2,\ t_1 b_1+t_2 b_2,\ t_1 c_1+t_2 c_2)}=0$ which, as we just computed, is singular. More precisely, the above map is undefined at the $9$ points where $F_{(a_1,\ b_1,\ c_1)} = F_{(a_2,\ b_2,\ c_2)} =0$. But, if we take $X$ to be $\mathbb{P}^2$ blown up at those $9$ points, then we get a map from the regular $X$ to the regular $S$, where every fiber is a <strike>nodal</strike> cuspidal cubic or worse.

Remark: Both of these counter-examples are still counter-examples if you replace "algebraically closed" by "perfect", but it would make my exposition messier.