Thanks to Igor Belegradek for his help in the comments. This answer is merely an expansion of his remarks.
The proof of the following proposition was adapted from the paper A simplification problem in manifold theory by Hausmann and Jahren, specifically Propositions $3.3$ and $3.11$.
Proposition: Let $M$ and $N$ be connected closed smooth manifolds such that $M\times\mathbb{R}$ and $N\times\mathbb{R}$ are diffeomorphic. Then $M$ and $N$ are smoothly $h$-cobordant.
Proof: Let $f : M\times\mathbb{R} \to N\times\mathbb{R}$ be a diffeomorphism. Note that $f^{-1}$ restricts to a diffeomorphism $(M\times\mathbb{R})\setminus f^{-1}(N\times\{0\}) \to N\times(-\infty, 0)\cup N\times(0,\infty)$. Let $A$ and $B$ be the connected components of $(M\times\mathbb{R})\setminus f^{-1}(N\times\{0\})$ with $B$ being the component containing points $(m, x)$ with $x$ arbitrarily large. By postcomposing $f$ with the map $N\times\mathbb{R} \to N\times\mathbb{R}$, $(n, x) \mapsto (n, -x)$ if necessary, we can assume that $f(A) = N\times(-\infty, 0)$ and $f(B) = N\times(0,\infty)$. Since $N$ is compact, the projection of $f^{-1}(N\times\{0\})$ to $\mathbb{R}$ is compact, so we can choose $s \in \mathbb{R}$ with $M\times\{s\} \subset B$ and hence $f(M\times\{s\}) \subset N\times (0, \infty)$. By the compactness of $M$, we can choose $\alpha > 0$ such that $f(M\times\{s\}) \subset N\times(0, \alpha)$.
Let $C = (N\times[0,\infty))\cap f(M\times(-\infty, s])$ be the bounded region between $N\times\{0\}$ and $f(M\times\{s\})$. This is a smooth cobordism between $N$ and $M$. We will prove that the inclusion $i: N \to C$, $n \mapsto (n, 0)$ is a homotopy equivalence. The same arguments can be used to show that $j: M \to C$, $m \mapsto f(m, s)$ is a homotopy equivalence (by considering $f^{-1}(C)$), so $C$ is in fact a smooth $h$-cobordism.
First note that the composition $N \xrightarrow{i} C \hookrightarrow N\times\mathbb{R} \xrightarrow{\operatorname{pr}_1} N$ is simply $\operatorname{id}_N$, so $i$ induces injective homomorphisms $i_* : \pi_k(N) \to \pi_k(C)$ for every $k$.
Now choose $t > s$ such that $f(M\times\{t\}) \subset N\times(\alpha, \infty)$. Let $C' = f(M\times[s, \infty))\cap (N\times(-\infty,\alpha])$ be the region bounded between $f(M\times\{s\})$ and $N\times\{\alpha\}$, and $C'' = (N\times[\alpha, \infty))\cap f(M\times(-\infty, t])$ be the region bounded between $N\times\{\alpha\}$ and $f(M\times\{t\})$.
Note that the composition $C \hookrightarrow C\cup C' = N\times[0,\alpha] \hookrightarrow C\cup C'\cup C'' = C\cup f(M\times[s, t]) \to C$ is $\operatorname{id}_C$, where the last map is a deformation retract. Therefore the map $\varphi : N\times[0,\alpha] \to C$ induces surjections on homotopy groups. Since $i : N \to C$ factorises as $N \xrightarrow{i_0} N\times[0,\alpha] \xrightarrow{\varphi} C$ where $i_0 : N \to N\times[0,\alpha]$, $n \mapsto (n, 0)$ is a homotopy equivalence, we see that $i_* : \pi_k(N) \to \pi_k(C)$ is surjective for every $k$.
So the inclusion $i : N\to C$ induces isomorphisms $i_* : \pi_k(N) \to \pi_k(C)$ for every $k$, and is therefore a homotopy equivalence. $\qquad\square$
In the language of the aforementioned paper, $C$, $C'$, and $C''$ are invertible cobordisms with $[C'] = [C]^{-1}$ and $[C''] = [C]$.
Theorem: Let $M$ be a smooth manifold homeomorphic to $T^n$ and suppose $M\times S^1$ is diffeomorphic to $T^{n+1}$. If $n \neq 4$, then $M$ is diffeomorphic to $T^n$.
Proof: Note that $\pi_1(M) \cong \mathbb{Z}^n$. For $n \leq 3$, it follows that $M$ is diffeomorphic to $T^n$ - see this question for the $n = 3$ case. Now suppose that $n \geq 5$.
Let $f : M\times S^1 \to T^{n+1}$ be a diffeomorphism. The covering space of $M\times S^1$ corresponding to $\pi_1(M)\times 1 \subset \pi_1(M\times S^1)$ is $M\times\mathbb{R}$, while $f_*(\pi_1(M)\times 1) \cong \mathbb{Z}^n$, so the corresponding covering space of $T^{n+1}$ is diffeomorphic to $T^n\times\mathbb{R}$. It follows that $M\times\mathbb{R}$ is diffeomorphic to $T^n\times\mathbb{R}$. By the Proposition above, there is a smooth $h$-cobordism between $M$ and $T^n$. Since the Whitehead group of $\mathbb{Z}^n$ is trivial and $n \geq 5$, the $h$-cobordism is diffeomorphic to a cylinder by the $s$-cobordism theorem, so $M$ is diffeomorphic to $T^n$. $\qquad\square$
This leaves only $n = 4$. It is not known if exotic tori exist in dimension four, but if they do, there may be infinitely many of them (as often happens in dimension four). If that were the case, there would be infinitely many non-diffeomorphic pairs $(M, N)$ of smooth manifolds homeomorphic to $T^4$ with $M\times S^1$ diffeomorphic to $N\times S^1$ (because there are only finitely many smooth manifolds homeomorphic to $T^5$). However, it does not necessarily follow that there are any such pairs with $N = T^4$.
Question (open?): Could there be an exotic four-torus such that its product with a circle is diffeomorphic to $T^5$?