(This should have been a comment to Andreas Blass' answer, but it did not fit there.) To answer the stronger question, asked in a comment to Andreas Blass' answer you can argue as follows in the case of a CW complex.

Suppose that $X$ is a CW complex, that $X$ is not simply connected, and that for any point $x$ in $X$ the space $X \setminus \{x\}$ is contractible, then $X$ is a circle.

If $X$ has cells in dimension at least three, then removing a point from the interior of such a cell does not change the 2-skeleton of the CW complex and hence does not affect the fundamental group (any homotopy between loops can be made to happen within the 2-skeleton). Since we are assuming that $X$ is not simply connected, but that the removal of any point makes the space contractible, it follows that $X$ cannot have cells of dimension three or more.

Similarly, removing a point in the interior of a cell of dimension two corresponds to removing a relation for the fundamental group of $X$. Again, since we are assuming that $X$ is not simply connected, the resulting space would have fundamental group surjecting to a non-trivial group and would therefore not be trivial. Therefore we deduce that $X$ has no cells of dimension two either.

We are left with $X$ having cells of dimension at most one. Thus $X$ is a wedge of circles and it is now easy to see that the stated condition implies that $X$ is in fact a single circle.

With similar arguments it seems that you can show also the following result. Suppose that $X$ is a CW complex such that for every point $x \in X$ the space $X \setminus \{x\}$ is contractible. Then either $X$ is itself contractible (e.g. $S^\infty$), or $X$ is homotopy equivalent (and maybe even homeomorphic) to a sphere.