I apologize for answering such an old question, but it seems fundamental. A classical counterexample occurs for the abel map of a Prym variety with exceptional singularities on the theta divisor. The point is that the fibers of the abel prym map X-->Y of the double cover C'-->C are included among those for the abel map of C', hence are all smooth. (A map obtained by restricting another map over a subvariety of the target has the same fibers.)
Nonetheless X is singular at any exceptional divisor. (see lemma 2.13 of A Riemann singularities theorem for Prym theta divisors, with applications).
The point of the previous paper was that generalizing the Riemann - Kempf singularity theorem to prym varieties is easy when X is smooth. But when X is singular it is considerably harder:
Singularities of the Prym theta divisor
For a detailed discussion of the case of the abel prym map for a prym variety isomorphic to the intermediate jacobian of the cubic threefold, see:
On parametrizing exceptional tangent cones to Prym theta divisors
The answer is yes however if the target Y is a smooth curve, since X is smooth at any point lying on a smooth cartier divisor, (compare Mumford, chap.7, Prop. 2, redbook.)
