@Ryan answers the question in three dimensions (and such a result cannot hold without the Poincare conjecture, since otherwise you could take a connected sum with a fake $S^3.)$ In general, this is a special case of the Borel Conjecture, which is known to hold in dimension four for groups of subexponential growth, such as $\mathbb{Z}^4$ (in the topological category). For more, see @Igor Belegradek's answer to [this question][1], and references therein.


  [1]: https://mathoverflow.net/questions/69431/more-four-dimensional-counterexamples