I apologize for answering such an old question, but it seems fundamental. A classical counterexample occurs for the abel map of a Prym variety with exceptional singularities on the theta divisor. The point is that the fibers of the abel prym map X-->Y of the double cover C'-->C are included among those for the abel map of C', hence are all smooth. (A map obtained by restricting another map over a subvariety of the target has the same fibers.)
Nonetheless X is singular at any exceptional divisor. (see lemma 2.13 of http://www.math.uga.edu/%7Eroy/sv2rst.pdfA Riemann singularities theorem for Prym theta divisors, with applications).
The point of the previous paper was that generalizing the Riemann - Kempf singularity theorem to prym varieties is easy when X is smooth. But when X is singular it is considerably harder:
http://www.math.uga.edu/%7Eroy/sv5rst2.pdfA necessary and sufficient condition for Riemann's singularity theorem to hold on a Prym theta divisor
http://annals.math.princeton.edu/2009/170-1/p05Singularities of the Prym theta divisor
For a detailed discussion of the case of the abel prym map for a prym variety isomorphic to the intermediate jacobian of the cubic threefold, see:
http://www.math.uga.edu/%7Eroy/onparam.pdfOn parametrizing exceptional tangent cones to Prym theta divisors
The answer is yes however if the target Y is a smooth curve, since X is smooth at any point lying on a smooth cartier divisor, (compare Mumford, chap.7, Prop. 2, redbook.)