When are those subgroups of $\mathrm{SL}(2, \mathbb{C})$ discrete?

Question

Let $A = \pmatrix{1 & 0 \\ \alpha & 1} $ and $ B = \pmatrix{1 & 1 \\ 0 & 1}$, where $\alpha \in \mathbb{C}$ is a complex parameter.

Now consider the family of representations $r_{\alpha}$ of the free group on two generators $F_2 = \langle a,b\rangle$ in $\mathrm{SL}(2, \mathbb{C})$ setting $r_{\alpha}(a) = A$ and $r_{\alpha}(b) = B$. One can see that when $\alpha$ is transcendental over $\mathbb{Q}$, the representation $r_{\alpha}$ is faithful (see T. Church & A. Pixton "Separating twists and the Magnus representation of the Torelli group" Lemma 5.1).

The question I am interested in is the following : when is (or is not) $r_{\alpha}(F_2)$ a discrete subgroup of $\mathrm{SL}(2, \mathbb{C})$ ?

I suppose this is a difficult question of dynamics, I am curious if anyone has ever studied similar questions.

Answer 1

The free discrete subgroups consist of the closure of the Riley slice of Schottky space (notice that one may assume $Re(\alpha)\geq 0, Im(\alpha)\geq 0$, since $\alpha \mapsto -\alpha, \overline{\alpha}$ preserves discreteness). Here's a picture of the first quadrant of the Riley slice:

The exterior of the black fractal represents free discrete groups that are generalized Schottky groups. The colored regions correspond to the combinatorics of the Ford domain. These have been investigated in detail by Akiyoshi, Sakuma, Wada, and Yamashita (see also their monograph where the picture was taken from). The black curve consists of geometrically finite groups with a cusp or degenerate groups. By the density conjecture, it is known that all free two-parabolic generator groups lie in the boundary of the Riley slice. Moreover, it is known that there is a group realizing each ending lamination parameterized by $\mathbb{R/Z}-\mathbb{Q/Z}$.

In the exterior of the Riley slice, there are many more non-free discrete groups, such as those corresponding to 2-bridge links. In fact, all of the torsion-free discrete groups correspond to 2-bridge links. Note that for 2-bridge links, although they are known to be generated by 2 parabolics, the precise value of $\alpha$ which gives the parabolics has not been determined. One may see an example of this for the twist knots, for which $\alpha$ has been worked out by Hoste and Shanahan.

For the discrete groups with torsion, there are certain other orbifolds closely related to the 2-bridge links; see some incomplete notes of mine. Conjecturally, all of the discrete groups are obtained by extending the "pleating locus" for each rational number $\in \mathbb{Q/Z}$ (representing a simple closed curve on the 4-punctured torus) through the cusp, and into the complement of Schottky space. The points where these elements are primitive elliptics should correspond to discrete groups (with some caveats); these pleating rays are displayed in the picture. In fact, the discrete groups in the complement of the Riley slice closure form a discrete set, with limit points at the boundary of the Riley slice.

Answer 2

If you want to know if your group is not discrete, apply Jorgensen's inequality (Jørgensen, Troels (1976), "On discrete groups of Möbius transformations", American Journal of Mathematics 98 (3): 739–749). It says that if $A$ and $B$ generate a non-elementary discrete subgroup of $SL_2(C)$, then $$ |\text{Tr}(A)^2 -4| + |\text{Tr}(ABA^{-1}B^{-1})-2|\ge 1. $$ For your group, this works out to $|\alpha|^2 \geq 1$. So for $|\alpha|<1$, your group is either elementary (eg if $\alpha=0$) or non-discrete.

In the case at hand, your matrices are parabolic, so you should be able to get the same result by an elementary analysis of fixed points on the sphere at infinity; this was actually Jorgensen's starting point (per the intro to his paper).

Answer 3

Yes, the answer is complicated, it is related to holomorphic dynamics, and the question was much studied, see for example: MR0869581 Lyubich, M. Yu.; Suvorov, V. V. Free subgroups of SL2(C) with two parabolic generators.

Answer 4

When $\alpha$ is very large one can see that the two elements play ping pong:

It is clear that if $\alpha ,n$ are large, $A,B^n$ play ping pong; so they are free. The proof of freeness also shows that the group generated by $A,B^n$ acts properly discontinuously on a piece of ${\mathbb P}^1({\mathbb C})$ and hence form a discrete subgroup of $SL_2({\mathbb C})$.

To reduce to the above situation, conjugate $A,B$ by a diagonal matrix so that $B$ is replaced by $B^n$ and $A$ is replaced by $A'= \begin{pmatrix}1 & 0 \cr \alpha /n &1\end{pmatrix}$ . If $n$ is large, and $\alpha /n$ is large, then by the preceding para, $A'$ and $B^n$ generate a discrete subgroup, hence so do the conjugates $A,B$.