The answer to your question is no. Here is a counterexample.

Let $X$ be the CW-complex obtained by attaching a 2-cell to the space $[-1,1]$ via the attaching map
$S^1\cong [-1,1]/\partial([-1,1]) \to [-1,1]$ given by (the continuous extension of) $x\mapsto x\sin(1/x)$.

Then $X\setminus \{0\}$ is homotopy equivalent to an infinite wedge of $S^1$'s.