Let me convert my comment to an answer: Your expectation is false at least if the dimension of your manifold is $\ge 5$. Indeed, in every odd dimension $m\ge 5$ Milnor (Corollary 12.9 and Example 12.14 in the reference below) constructs examples of pairs of smooth compact manifolds $(L_1, L_2)$ and smooth $h$-cobordisms between these manifolds such that $L_1, L_2$ are not homeomorphic. Now, consider two pairs $(L_1, L_2), (L_2, L_1)$ and $h$-cobordisms $W_1, W_2$ between these pairs which are the same apart from the orientation. In particular, they have opposite torsions. Now, glue $W_1, W_2$ along the copies of $L_2$ via the identity map. The result is an $h$-cobordism from $L_1$ to $L_1$ with zero torsion. Hence, by the $s$-cobordism theorem, $W$ is diffeomorphic to the product $L_1\times [0,1]$. Then, glue together infinitely many such $h$-cobordisms. The resulting manifold $N$ will be diffeomorphic to $L_1\times \mathbb R$. At the same time, $N$ is also a union of trivial $s$-cobordisms between copies of $L_2$, hence, is diffeomorphic to $L_2\times \mathbb R$. Hence, $L_1\times \mathbb R$ is diffeomorphic to $L_2\times \mathbb R$, but $L_1$ is not homeomorphic to $L_2$.
Milnor, John W., Whitehead torsion, Bull. Am. Math. Soc. 72, 358-426 (1966). ZBL0147.23104.
A famous open problem along the lines of your question is: Suppose that $M_1, M_2$ are compact aspherical topological manifolds such that $M_1\times \mathbb R$ is homeomorphic to $M_2\times \mathbb R$. Is it true that $M_1$ is homeomorphic to $M_2$?