Return to Answer

The answer by Misha is excellent. Let me however give a more concise and self contained version, which the OP could use when writing his paper.

First note that the statement is true for $G$ iff it is true for $\{\alpha g\mid g\in G,~\alpha\in \mathbb{R}^*\}$. Since $\text{SL}_3(\mathbb{R})\to \text{PGL}_3(\mathbb{R})$ is onto, it is enough to prove:

$(*)$ Let $G<\text{SL}_3(\mathbb{R})$ be a subgroup such that all eigenvalues of all elements are of absolute value 1. Then $G$ is conjugated to $\text{SO}(3)$.

Note that for $g\in G$ the eigenvalues are $1,\alpha,\bar{\alpha}$ for some $\alpha\in S^1\subset\mathbb{C}$. In particular $\text{tr}(g)=\text{tr}(g^{-1})$. It follows that $G$ is not Zariski dense. Its Zariski closure must be reductive by the irreducibility assumption, so it is contained in a conjugate of $\text{SO}(3)$ or $\text{SO}(2,1)$ - these are the irreducible reductive subgroups. We are left to contradict the second possibility.

Recalling that the connected component of $\text{SO}(2,1)$ is isomorphic to $\text{SL}_2(\mathbb{R})$ and by passing to a finite index subgroup of $G$, let me deal now with statement $(*_2)$, the 2-dim analogue of $(*)$.

I will explain why if $G<\text{SL}_2(\mathbb{R})$ is Zariski dense then it is also dense wrt the usual topology, which will give a contradiction.

Assume $G$ is Zariski dense. Find $g\in G$ of infinite order. Observe that $g$ is not unipotent. Indeed, otherwise a generic conjugate of it $h\in G$ will be another unipotent and the product of high powers $g^nh^m$ will have $\text{tr}>2$ (this becomes clear when you find a basis in which $g,h$ are represented in upper,lower triangular matrix forms). It follows that $g$ is eliptic, hence, up to a choice of basis, $g\in\text{SO}(2)$ is an irrational rotation. It folows that $\text{SO}(2)<\bar{G}$. Since $\text{SO}(2)$ is a maximal subgroup, $\bar{G}=\text{SL}_2(\mathbb{R})$.

The answer by Misha is excellent. Let me however give a more concise and self contained answer, which the OP could use when writing his paper.

First note that the statement is true for $G$ iff it is true for $\{\alpha g\mid g\in G,~\alpha\in \mathbb{R}^*\}$. Since $\text{SL}_3(\mathbb{R})\to \text{PGL}_3(\mathbb{R})$ is onto, it is enough to prove:

$(*)$ Let $G<\text{SL}_3(\mathbb{R})$ be a subgroup such that all eigenvalues of all elements are of absolute value 1. Then $G$ is conjugated to $\text{SO}(3)$.

Note that for $g\in G$ the eigenvalues are $1,\alpha,\bar{\alpha}$ for some $\alpha\in S^1\subset\mathbb{C}$. In particular $\text{tr}(g)=\text{tr}(g^{-1})$. It follows that $G$ is not Zariski dense. Its Zariski closure must be reductive by the irreducibility assumption, so it is contained in a conjugate of $\text{SO}(3)$ or $\text{SO}(2,1)$ - these are the irreducible reductive subgroups. We are left to contradict the second possibility.

Recalling that the connected component of $\text{SO}(2,1)$ is isomorphic to $\text{SL}_2(\mathbb{R})$ and by passing to a finite index subgroup of $G$, let me deal now with statement $(*_2)$, the 2-dim analogue of $(*)$.

I will explain why if $G<\text{SL}_2(\mathbb{R})$ satisfying the conditions above is Zariski dense then it is also dense wrt the usual topology, which will give a contradiction.

Assume $G$ is Zariski dense. Find $g\in G$ of infinite order. Observe that $g$ is not unipotent. Indeed, otherwise a generic conjugate of it $h\in G$ will be another unipotent and the product of high powers $g^nh^m$ will have $\text{tr}>2$ (this becomes clear when you find a basis in which $g,h$ are represented in upper,lower triangular matrix forms). It follows that $g$ is eliptic, hence, up to a choice of basis, $g\in\text{SO}(2)$ is an irrational rotation. It folows that $\text{SO}(2)<\bar{G}$. Since $\text{SO}(2)$ is a maximal subgroup, $\bar{G}=\text{SL}_2(\mathbb{R})$.

You should not worry about scalars: this is true for $G$ iff it is true for $\{\alpha g\mid g\in G,~\alpha\in \mathbb{R}^*\}$. Since $\text{SL}_3(\mathbb{R})\to \text{PGL}_3(\mathbb{R})$ is onto, it is enough to prove:

Let $G<\text{SL}_3(\mathbb{R})$ be a subgroup such that all eigenvalues of all elements are of absolute value 1. Then $G$ is conjugated to $\text{SO}(3)$.

Since the condition is (real) algebraic (this is not hard to see), you can assume that $G$ is Zariski closed. By the irreducibility assumption the identity connected component is semisimple and by KAK decomposition it must be compact. It follows that $G$ is compact, hence conjugated into a maximal compact subgroup, namely $\text{SO}(3)$.

BTW, the above works just as well for sub-semi-groups of $\text{GL}_3(\mathbb{R})$, because the Zariski closure of such a semi-group is a group.

The answer by Misha is excellent. Let me however give a more concise and self contained version, which the OP could use when writing his paper.

First note that the statement is true for $G$ iff it is true for $\{\alpha g\mid g\in G,~\alpha\in \mathbb{R}^*\}$. Since $\text{SL}_3(\mathbb{R})\to \text{PGL}_3(\mathbb{R})$ is onto, it is enough to prove:

$(*)$ Let $G<\text{SL}_3(\mathbb{R})$ be a subgroup such that all eigenvalues of all elements are of absolute value 1. Then $G$ is conjugated to $\text{SO}(3)$.

Note that for $g\in G$ the eigenvalues are $1,\alpha,\bar{\alpha}$ for some $\alpha\in S^1\subset\mathbb{C}$. In particular $\text{tr}(g)=\text{tr}(g^{-1})$. It follows that $G$ is not Zariski dense. Its Zariski closure must be reductive by the irreducibility assumption, so it is contained in a conjugate of $\text{SO}(3)$ or $\text{SO}(2,1)$ - these are the irreducible reductive subgroups. We are left to contradict the second possibility.

Recalling that the connected component of $\text{SO}(2,1)$ is isomorphic to $\text{SL}_2(\mathbb{R})$ and by passing to a finite index subgroup of $G$, let me deal now with statement $(*_2)$, the 2-dim analogue of $(*)$.

I will explain why if $G<\text{SL}_2(\mathbb{R})$ is Zariski dense then it is also dense wrt the usual topology, which will give a contradiction.

Assume $G$ is Zariski dense. Find $g\in G$ of infinite order. Observe that $g$ is not unipotent. Indeed, otherwise a generic conjugate of it $h\in G$ will be another unipotent and the product of high powers $g^nh^m$ will have $\text{tr}>2$ (this becomes clear when you find a basis in which $g,h$ are represented in upper,lower triangular matrix forms). It follows that $g$ is eliptic, hence, up to a choice of basis, $g\in\text{SO}(2)$ is an irrational rotation. It folows that $\text{SO}(2)<\bar{G}$. Since $\text{SO}(2)$ is a maximal subgroup, $\bar{G}=\text{SL}_2(\mathbb{R})$.

You should not worry about scalars: this is true for $G$ iff it is true for $\{\alpha g\mid g\in G,~\alpha\in \mathbb{R}^*\}$. Since $\text{SL}_3(\mathbb{R})\to \text{PGL}_3(\mathbb{R})$ is onto, it is enough to prove:

Let $G<\text{SL}_3(\mathbb{R})$ be a subgroup such that all eigenvalues of all elements are of absolute value 1. Then $G$ is conjugated to $\text{SO}(3)$.

Since the condition is (real) algebraic, you can assume that $G$ is Zariski closed. By the irreducibility assumption the identity connected component is semisimple and by KAK decomposition it must be compact. It follows that $G$ is compact, hence conjugated into a maximal compact subgroup, namely $\text{SO}(3)$.

BTW, the above works just as well for sub-semi-groups of $\text{GL}_3(\mathbb{R})$, because the Zariski closure of such a semi-group is a group.

You should not worry about scalars: this is true for $G$ iff it is true for $\{\alpha g\mid g\in G,~\alpha\in \mathbb{R}^*\}$. Since $\text{SL}_3(\mathbb{R})\to \text{PGL}_3(\mathbb{R})$ is onto, it is enough to prove:

Let $G<\text{SL}_3(\mathbb{R})$ be a subgroup such that all eigenvalues of all elements are of absolute value 1. Then $G$ is conjugated to $\text{SO}(3)$.

Since the condition is (real) algebraic (this is not hard to see), you can assume that $G$ is Zariski closed. By the irreducibility assumption the identity connected component is semisimple and by KAK decomposition it must be compact. It follows that $G$ is compact, hence conjugated into a maximal compact subgroup, namely $\text{SO}(3)$.

BTW, the above works just as well for sub-semi-groups of $\text{GL}_3(\mathbb{R})$, because the Zariski closure of such a semi-group is a group.