Here is a first counter-example.
Suppose that $V$ is a genus two handlebody of dimension three. For example, $V$ can be obtained by embedding a “eye-glasses” graph in three-space and taking a small regular neighbourhood. Let $M$ be the double of $V$ across its boundary. That is, we take two copies of $V$ and glue via the identity on the boundary.
The universal cover of $M$ is a copy of the three-sphere, minus a Cantor set.