Here is what seems to be a counterexample. Let (M,g) be a simply-connected closed Riemannian manifold. Then M times (0,infinity) with the warped product metric dr^2 + r^2 g has bounded curvature and the completion at r=0 is a point. If the metric is smooth, then M is diffeomorphic to a sphere, so any other M gives a counterexample.
EDIT: Sorry, this does not work as curvature blows up at zero unless g has constant curvature 1.