This is false for $n=1.$ The mapping class group of the torus is $SL(2, Z),$ of which the homeomorphisms you describe are but a small part - the parabolic matrices $\begin{pmatrix}1 & n\\ 0 &1\end{pmatrix},$ unless I am very confused. I cautiously believe the statement is true for $n=2,$ by
Allen Hatcher, MR 420620 Homeomorphisms of sufficiently large $P^{2}$-irreducible $3$-manifolds, Topology 15 (1976), no. 4, 343--347.
UPDATE for $n=3,$ the best I can find is:
Richard Stong and Zhenghan Wang, MR 1769331 Self-homeomorphisms of 4-manifolds with fundamental group Z, Topology Appl. 106 (2000), no. 1, 49--56.
Which classifies up to pseudo-isotopy (NOT isotopy), and does give the expected result, as far as I can tell.