There are indeed counterexamples to which Igor Belegradek gave reference. Here is another counterexample in the plane, perhaps the simplest there is: Let $X$ be an annulus with one arc attached to one of its boundary components and another arc attached to the other boundary component, and $Y$ -  an annulus with two disjoint arcs attached to the same one of its boundary components.

[![enter image description here][1]][1]


  [1]: https://i.sstatic.net/ljLSo.jpg