I apologize for answering such an old question, but it seems fundamental. A classical counterexample occurs for the abel map of a Prym variety with exceptional singularities on the theta divisor. The point is that the fibers of the abel prym map of the double cover C'-->C are included among those for the abel map of C', hence are all smooth. (A map obtained by restricting another map over a subvariety of the target has the same fibers.)
Nonetheless X is singular at any exceptional divisor. (see lemma 2.13 of http://www.math.uga.edu/%7Eroy/sv2rst.pdf) For a detailed discussion of the case of the abel prym map for a prym variety isomorphic to the intermediate jacobian of the cubic threefold, see:
http://www.math.uga.edu/%7Eroy/onparam.pdf
The answer is yes however if the target Y is a smooth curve, since X is smooth at any point lying on a smooth cartier divisor, (compare Mumford, chap.7, Prop. 2, redbook.)