I will consider finitely generated subgroups G of $SL(3,R)$ as it suffices. Since all eigenvalues are assumed to be on the unit circle you have a dichotomy.

The subgroup G contains an elliptic element of infinite order. Then the subgroup I'd not discrete. Take its topological closure. It is a Lie subgroup of positive dimension. Consider the identity component. By inspection, it is either reducible, and so is your subgroup, or it is irreducible compact, or it contains a subgroup locally isomorphic to $SL(2,R)$. In the latter case the original group contains an element with an eigenvalue not on the unit circle. If the compact subgroup us irreducible, then G itself is relatively compact. In any case, you are done.

All eigenvalues are roots of unity. Since G is finitely generated, it contains a finite index subgroup where all elements are unipotent. Thus finite index a subgroup is contained in upper triangular group (up to conjugation). Then you are done again.

**Edit.** Here are some details. Below and above $R={\mathbb R}$ and $C={\mathbb C}$.

Most ingredients of the proof follow closely the proof of the Tits’ alternative as presented in our book) with Drutu, except few things are easier since you are dealing with subgroups of $SL(3,R)$.

First, some terminology: A subgroup $G$ of $SL(n,C)$ is called **distal** if all eigenvalues of all elements of $G$ have absolute value 1 (such matrices are also called **distal**). An element of $SL(n,C)$ is called **quasiunipotent** if all its eigenvalues are roots of unity.

Here is an elementary lemma that I will use repeatedly:

Lemma 1. If $N< SL(3,R)$ is a nontrivial connected nilpotent Lie subgroup then $N$ either preserves a unique line $L$ or a unique plane $P$ in $R^3$. In particular, if a group $G$ normalizes $N$ then $G$ also preserves that line/plane.

Below, $G< SL(3,R)$ is distal and finitely generated.

Case 1. All its elements of $G$ are quasiunipotent.

One can shorten a bit the next chain of lemmas if one is willing to use Borel’s theorem that every finitely generated subgroup $G$ of $SL(n,R)$ contains a **neat subgroup** $G_0$ of finite index, i.e. a subgroup such that each quasiunipotent element of $G_0$ is unipotent. Assuming this, we see that if all elements of $G$ are quasiunipotent, then $G$ contains a finite index subgroup $G_1<G$ such that all elements of $G_1$ are unipotent. Hence, by Kolchin’s theorem, $G_0$ is conjugate to the group of upper triangular matrices.

Instead of relying upon Borel and Kolchin's theorems, one can give a direct argument (the proofs are elementary, you can find them in our book for instance):

Lemma 2. If $G< GL(n,C)$ is a finitely generated subgroup such that all elements of $G$ are quasiunipotent then the orders of roots of unity appearing as eigenvalues are uniformly bounded above.

Lemma 3. If $G< GL(n,C)$ is a finitely generated subgroup such that all elements of $G$ are quasiunipotent with uniformly bounded orders of roots of unity appearing as eigenvalues, then $G$ contains a finite index subgroup $G_1$ which is conjugate to the group of upper triangular matrices. (The proof of this lemma is similar to the proof of Kolchin’s theorem.)

Corollary. If $G< GL(n,C)$ is finitely generated and each element of $G$ is quasiunipotent then $G$ is virtually nilpotent.

Now, if $G< SL(3,R)$ and all its elements are quasiunipotent, then, using Lemma 1, we see that $G$ is either finite or preserves a proper subspace of $R^3$.

Case 2. Suppose that $G< SL(3,R)$ is distal, but some $g\in G$ is not quasiunipotent. By looking at the complex Jordan normal form of such $g$ you see that it has to be diagonal (over ${\mathbb C}$), which means that $g$ is **elliptic**, i.e. is conjugate to a rotation $h\in SO(3)$; and this rotation has infinite order. (Here the argument is simplified by the fact that we have $3\times 3$ matrices.) Therefore, the group $G$ cannot be discrete. Consider its closure $cl(G)$ (in the classical topology) in $SL(3,R)$. Let $H$ denote the identity component of the closure (which has to be a Lie subgroup of positive dimension). The subgroup $H$ is normalized by $G$. Let $N< H$ be the nilpotent radical. Since it is a characteristic subgroup, it is also normalized by $G$. Therefore, by Lemma 1, $G$ preserves a proper subspace in $R^3$. Thus, assume that $H$ is reductive. Again, by appealing to Lemma 1 we see that $H$ has to be semisimple.

There are not that many semisimple proper connected subgroups in $SL(3,R)$, they are are $SO(2,1)$ and $SL(2,R)$ and $SO(3)$ (up to conjugation).

If $H\cong SO(3)$ then it preserves a unique inner product on $R^3$, from which it follows that $G$ also preserves the same inner product and, hence, $G$ is relatively compact in this case. Suppose that $H$ is locally isomorphic to $SL(2,R)$ or $SL(3,R)$. By density of $G$ in $H$, it follows that $G$ is not distal, a contradiction.

Lastly, here is how one can handle the case of subgroups $G$ which are not finitely generated. Represent $G$ as a union of finitely generated subgroups $G_i$, $i\in I$.

(a) If the closure of some $G_i$ contains a nontrivial semisimple subgroup $K$, this subgroup has to be compact. Then you can see that the closure of each $G_j$ containing $G_i$ also contains $K$, from which you can see that each $G_j$ preserves the unique $K$-invariant inner product. Hence, $G$ is relatively compact.

(b) Assume that $G$ is not relatively compact. Then there exists $G_i$ which preserves a unique proper subspace $V$ in $R^3$. Since each $G_j$ containing $G_i$ also preserve a unique proper subspace, $V$ has to be invariant under each $G_j$ and, thus, $V$ is $G$-invariant. qed

Last remark: There are examples of finitely generated subgroups $G$ of $SL(n,C)$ such that each element of $G$ is elliptic (conjugate to a unitary matrix) but $G$ itself is not relatively compact. This is something to be aware of.

5more comments