van der Kallen gave a nice answer for the situation at hand, but I thought I'd give a somewhat more general one. The question is equivalent to showing that $(\mathbb{R}^n)_G=0$ for some finite-index torsion-free subgroup $G$ of $\text{GL}_n(\mathbb{Z})$. The representation $\mathbb{R}^n$ is a nontrivial irreducible representation of $\text{SL}_n(\mathbb{R})$, so this follows from the following more general result (just take $\Gamma$ to be a torsion-free lattice, for instance the level $3$ principle congruence subgroup).

LEMMA : Let $V$ be a nontrivial finite-dimensional irreducible representation of $\text{SL}_n(\mathbb{R})$ over $\mathbb{R}$ and let $\Gamma$ be any lattice in $\text{SL}_n(\mathbb{R})$. Then $V_{\Gamma} = 0$.

To see this, observe that using the Borel density theorem (which says that $\Gamma$ is Zariski dense in $\text{SL}_n(\mathbb{R})$), we can get that $V$ is also a nontrivial irreducible $\Gamma$-representation. Now, $V_{\Gamma} = V/K$ where $K$ is spanned by the set $\{x-g(x)\text{ $|$ }x \in V, g \in \Gamma\}$. Clearly $K$ is a nontrivial $\Gamma$-subrepresentation of $V$, so by the irreducibility of $V$ we must have $K=V$.