Here is a first counter-example. 

Suppose that $V$ is a genus two handlebody of dimension three.  For example, $V$ can be obtained by embedding a “eye-glasses” graph in three-space and taking a small regular neighbourhood. Let $M$ be the *double* of $V$ across its boundary.  That is, we take two copies of $V$ and glue via the identity on the boundary.

The universal cover of $M$ is a copy of the three-sphere, minus a Cantor set.