Actually, the definition you gave in the post differs from the one in the book. First of all, you should apply $\iota$, where $\iota(\varphi)(x)=\varphi(-x)$, after the inverse Fourier transform. Secondly,The test function $\varphi$ should lie in $\mathcal S(\mathbb Z^n)$, not in $C^\infty(\mathbb T^n)$, since the $\mathcal F_{\mathbb T^n}$ maps the second of these spaces into the first, so for $\varphi\in C^\infty(\mathbb T^n)$ the expression $\mathcal F_{\mathbb T^n}^{-1}\varphi$ is undefined, as you correctly noticed. So, the correct formula would be $$ \langle\mathcal F_{\mathbb T^n}u,\varphi \rangle=\langle u,\iota\,\circ\,\mathcal F_{\mathbb T^n}^{-1}\varphi \rangle. $$$$ \langle\mathcal F_{\mathbb T^n}u,\varphi \rangle=\langle u,\iota\,\circ\,\mathcal F_{\mathbb T^n}^{-1}\varphi \rangle, $$ where $\iota(f)(x)=f(-x)$ and $\varphi \in \mathcal S(\mathbb Z^n)$.