Can you show any finite subgroup of $SL_2(R)$ is cyclic without using an invariant form?

3$\begingroup$ Of course  just think about the action on the hyperbolic plane. $\endgroup$ – HJRW Sep 23 '12 at 14:18

1$\begingroup$ Of course, a fixed point in the hyperbolic plane is the same as an invariant positivedefinite bilinear form. $\endgroup$ – Misha Sep 23 '12 at 15:31

1$\begingroup$ Why do you want to do this? More specifically, the proofs that any compact (e.g., finite) subgroup of a connected Lie group lies in a maximal compact subgroup and the maximal compact subgroups are pairwise conjugate rest on the idea of invariant forms (through the perspective of a fixedpoint theorem, which in turn inspired the fundamental Borel fixedpoint theorem in the algebraic theory), so it is both fruitful and natural to use invariant forms. And very efficient too. $\endgroup$ – grp Sep 23 '12 at 18:38

$\begingroup$ If the group is abelian then since every element is diagonalizable, the group is digonalizable over $C$ so the group is a subgroup of $C^*$. Now how to deal with the nonabelian case? $\endgroup$ – i. m. soloveichik Sep 23 '12 at 23:20

$\begingroup$ @solovei: As HW and Igor told you: Look at the fixed point of the action on the hyperbolic plane and, hence, reduce the problem to the case of subgroups of SO(2), which, I assume you know how to handle. $\endgroup$ – Misha Sep 24 '12 at 4:03
Here's one approach with no geometry/integration, using finite groups. First prove it for a (nontrivial) finite $p$group. If $p$ is odd, it contains a central element of order $p$, whose centralizer is $\mathbf{C}$diagonalizable, hence embeddable as a $p$subgroup of the nonzero complexes; this has to be cyclic. For $p=2$, it contains $I$, and you need to proceed with the same argument in $\text{PSL}(2,\mathbf{R})$, so your group stands as an extension with order 2 kernel and cyclic quotient (of order some power of 2), with only one element of order 2: lifting a generator from the quotient we obtain a group which is either cyclic or direct product of cyclic with $\{I,I\}$; since there's only one element of order 2, we deduce that that such a group is cyclic. So all Sylow subgroups are cyclic: this was the easy part.
Now a group whose Sylow subgroups are cyclic is called a Zgroup and these are known, apparently by Zassenhaus in the 30's, that such a group is metacyclic. I don't know how hard is this result, but it is certainly not relying on the Classification, nor any deep part of it.
So let's go ahead with a (nontrivial) metacyclic group $G$. It contains a nontrivial normal cyclic subgroup $H$, with cyclic quotient. If $H$ has order 2, the lifting argument above shows that the group is either cyclic or direct product of a cyclic group of order 2 and another cyclic group, and in this case, the latter has to be of odd order, so the whole group is cyclic anyway. Otherwise, $H$ has order $\ge 3$, and hence preserves exactly two lines in $\mathbf{R}^2$. So the normalizer $N$ of $H$ in $\text{GL}_2(\mathbf{C})$ preserves this unordered pair of lines. Let $Z$ be the centralizer of $H$ in $\text{GL}_2(\mathbf{C})$, which is abelian and has index 2 in $N$. Let us check that any two elements $g,h$ in $G\cap (NZ)$ commute. Observe that all elements in $NZ$ are square roots of $I$. It follows that $g$ conjugates $gh$ to its inverse $h^{1}g^{1}$ (which is equal to $hg^{1}=g^2hg^{1}$). Now if $x$ is a element in $\text{SL}_2(\mathbf{R})$ of finite order $\ge 3$, its centralizer in $\text{GL}_2(\mathbf{R})$ is contained in $\text{SL}_2(\mathbf{R})$, and it is conjugate to its inverse by some element not in $\text{SL}_2(\mathbf{R})$; it follows that $x$ is not conjugate to its inverse in $\text{SL}_2(\mathbf{R})$. Thus $gh$ has order at most 2 and $g$ and $h$ commute. Thus either $G\subset Z$ and $G$ is commutative, or $G$ is generated by $G\smallsetminus Z$ which is a commutative subset and hence $G$ is commutative anyway. So $G$ is abelian and we're done.
Edit: Here's a way to have a selfcontained approach, avoiding the use of Zassenhaus's theorem.
Taking a minimal counterexample, we get a finite group all of whose maximal subgroups are cyclic.
Assume that a finite group $G$ is not abelian and has all its maximal subgroups abelian; let us show that $G$ is abelianbycyclic. It amounts to proving that one of its maximal subgroups is abelian. Its center cannot be maximal (this holds in any group!). So any maximal subgroup contains a noncentral element. Since the maximal subgroup is abelian, it is actually the centralizer of any of its noncentral elements. Besides, any maximal subgroup contains the center $Z$ of $G$ (otherwise $G$ would be abelian). It follows that $M\cap N=Z$ for any two distinct maximal subgroups. Let $(M_i)_{1\le i\le k}$ representatives of conjugacy classes of maximal subgroups. Let $r_i$ be the index of the normalizer of $M_i$, so that $M_i$ has exactly $r_i$ conjugates. Since the maximal subgroups minus $Z$ partition $G\smallsetminus Z$, and setting $z=Z$ and $m_i=M_i$, $g=G$, we get $z+\sum_i (m_iz)r_i=g$. Assume by contradiction that for all $i$, $M_i$ is not normal; thus $m_ir_i=g$ for all $i$, and we get $(k1)g=z(1+\sum_i r_i)$. If $M_i$ is the centralizer of some $x_i$, then $r_i$ is the cardinal of the conjugacy class of $x_i$. These conjugacy classes are disjoint modulo $Z$, hence $\sum r_i\le g/z$, hence $k<2$. So $k=1$ and the above equation then reads $z(1+r_1)=0$, so $r_1=1$, a contradiction, so $G$ is abelianbycyclic.
Turning back to our minimal counterexample, it is thus metacyclic. The first and third part of the previous argument, which do not rely on the Zassenhaus theorem, show that any metacyclic subgroup of $\text{SL}_2(\mathbf{R})$ is cyclic.

$\begingroup$ I am confused by the claim in the third paragraph of the original argument that all elements of $N \setminus Z$ have order two. Consider the cyclic subgroup $H$ of $GL_2({\mathbb C})$ generated by a diagonal matrix whose diagonal entries are the two primitive cube roots of $1$. Let $x$ be a $2 \times 2$ matrix whose diagonal elements are zero and whose off diagonal elements are $1$ and $1$. Then $x$ has order four and normalizes but does not centralize $H$. Maybe I am missing something. If I am, could you finish instead by noting that $SL_2({\mathbf R})$ has one element of order two? $\endgroup$ – John Shareshian Sep 25 '12 at 14:25

$\begingroup$ you're right, they're order 4 (if you diagonalize $H$ over complex, it's antidiagonal matrices of determinant 1). I edited accordingly. $\endgroup$ – YCor Sep 25 '12 at 15:08

$\begingroup$ So I think your argument might work over any field (of $char\ne 0$) so that the roots of unity in the field is just $\pm 1$. N'est pas? $\endgroup$ – i. m. soloveichik Sep 26 '12 at 0:44
Here is an answer using character theory. For an irreducible character $\chi$ of a finite group $G$, define $\nu_2(\chi)$ to be $1$ if $\chi$ is afforded by a real representation, $0$ if $\chi$ is not real valued, and $1$ otherwise. Let $t$ be the number of elements of order two in $G$. As shown in Chapter 4 of Martin Isaacs' ``Character Theory of Finite Groups", the sum of $\nu_2(\chi)\chi(1)$ over all irreducibles of $G$ is $1+t$.
Now assume for contradiction that $G$ is a counterexample of minimal order to the claim that every finite subgroup of $SL_2({\mathbb R})$ is cyclic. So, every proper subgroup of $G$ is cyclic. (I believe that such groups were classified in a 1903 paper of Miller and Moreno, but we will not need that.)
As a group of odd order has no irreducible character of even degree and every reducible finite subgroup of $SL_2({\mathbb C})$ is cyclic, $G$ is even and and $G$ acts irreducibly on ${\mathbb C}^2$. As $SL_2({\mathbb R})$ has a unique element of order two, so does $G$. Therefore, if $G$ is a $2$group, then $G$ is quaternion of order eight, as follows from an analysis of $2$groups with a cyclic subgroup of index two. However, the group $Q_8$ has four linear characters, each of which satisfies $\nu_2(\chi)=1$. Since $Q_8$ has one element of order two, the unique $2$dimensional irreducible of $Q_8$ must satisfy $\nu_2(\chi)=1$, contradicting our assumption.
We assume now that $G$ is not a $2$group. A Sylow $2$subgroup $S$ of $G$ is cyclic by our minimality assumption. Thus $Aut(S)$ is a $2$group and $S$ is central in its normalizer $N_G(S)$. By Burnside's normal $p$complement theorem, $G$ has a normal subgroup $N$ of index $S$. We know that $N$ is cyclic. $S$ normalizes every Sylow subgroup of $N$. Since $G$ is not cyclic, $S$ does not centralize $N$ and therefore there is some Sylow $p$subgroup $P$ of $N$ that is not centralized by $S$. By minimality of $G$, $G=SP$. An analysis of $Aut(P)$ shows that $S$ does not centralize the unique subgroup of order $p$ in $P$. By minimality of $G$, $P=p$.
Some element $s$ of $S$ acts as an automorphism of order two on $P$ and therefore inverts a generator of $P$. By minimality of $G$, $S=\langle s \rangle$. Now $s^2$ is central in $G$ and $\langle s^2,P \rangle$ is abelian and has index two in $G$. Therefore, every irreducible of $G$ has degree at most two. If $s=2$ then $G$ is dihedral and has $p$ elements of order two, giving a contradiction. If $s>4$ then $G$ cannot have a faithful real irreducible character $\chi$ of degree two, as the image of $s^2$ must be a scalar matrix in any irreducible representation of $G$. Therefore, $S=4$.
Since $[G,G]=P$, $G$ has four linear characters and $p1$ irreducible characters of degree two, as the sum of the squares of the degrees of the irreducible characters is $G=4p$. Two of the linear characters satisfy $\nu_2(\chi)=1$ and the other two satisfy $\nu_2(\chi)=0$.
The quotient $G/\langle s^2 \rangle$ is dihedral of order $2p$ and therefore has $(p1)/2$ characters of degree two, each of which satisfies $\nu_2(\chi)=1$ (either by direct construction or by counting involutions). Each of these characters lifts to a (nonfaithful) irreducible of $G$. Since $G$ has a unique element of order two, the remaining $(p1)/2$ degree two irreducibles of $G$ must satisfy $\nu_2(\chi)=1$. Therefore, $G$ has no faithful real irreducible representation and our proof is complete.

$\begingroup$ @John: you were correct that my proof was incomplete. It would have been possible to fix it, but essentially along the lines you did, so there seems little point, given that you have already done it. $\endgroup$ – Geoff Robinson Sep 24 '12 at 20:20

$\begingroup$ Thanks, Geoff. I deleted all references to your answer. I hope that's OK. $\endgroup$ – John Shareshian Sep 25 '12 at 14:28
To show that any subgroup $\Gamma$ of $PSL(2, \mathbb{R})$ is cyclic, you can look at the quotient orbifold. If it has topology (that is, at least one handle), it is infinite (take a loop going around the handle). If it does not, and has no orbifold points, the group is trivial, since non topology>simply connected, and there are no connected nontrivial covers. If exactly one orbifold point, the group is cyclic. If the group has two orbifold points, the figure 8 around the two orbifold points has infinite order.
Lifting to $SL(2, \mathbb{R})$ is left as an exercise for the reader.
Of course, using the observation that any action on a negatively curved space has a fixed point, using convexity of distance functions (which is basically the invariant form argument you are alluding to) is much more general/robust argument.