I apologize for answering such an old question, but it seems fundamental.  A classical counterexample occurs for the abel map of a Prym variety with exceptional singularities on the theta divisor.  The point is that the fibers of the abel prym map  X-->Y of the double cover C'-->C are included among those for the abel map of C', hence are all smooth.  (A map obtained by restricting another map over a subvariety of the target has the same fibers.)  

Nonetheless X is singular at any exceptional divisor. (see lemma 2.13 of http://www.math.uga.edu/%7Eroy/sv2rst.pdf).  

The point of the previous paper was that generalizing the Riemann - Kempf singularity theorem to prym varieties is easy when X is smooth.  But when X is singular it is considerably harder:

http://www.math.uga.edu/%7Eroy/sv5rst2.pdf

http://annals.math.princeton.edu/2009/170-1/p05


For a detailed discussion of the case of the abel prym map for a prym variety isomorphic to the intermediate jacobian of the cubic threefold, see:

http://www.math.uga.edu/%7Eroy/onparam.pdf

 The answer is yes however if the target Y is a smooth curve, since X is smooth at any point lying on a smooth cartier divisor,  (compare Mumford, chap.7, Prop. 2, redbook.)