## Does such an infinite index subgroup exist?

Notation: If $G$ is a countable group and $H$ is a subgroup, for $g\in G$, let $|\mathcal{O}_{gH}|$ be the size of the $H$-orbit of $gH$ in the $H$-set $G/H$.

Does there exist a countable group $G$ and a subgroup $H$ with $[G\colon H]=\infty$ such that:

1. There is a $g\in G$ with $|\mathcal{O} _{gH}|\neq |\mathcal{O} _{g^{-1}H}|$,
2. $|\mathcal{O} _{gH}|=\infty$ if and only if $|\mathcal{O} _{g^{-1}H}|=\infty$, and
3. There is a constant $M>0$ such that $\displaystyle \frac{|\mathcal{O} _{gH}|}{|\mathcal{O} _{g^{-1}H}|} \leq M$ for all $g\in G$ with $|\mathcal{O} _{gH}|,|\mathcal{O} _{g^{-1}H}|\neq \infty$.

For instance, 2 and 3 above would be satisfied if:

• If $|\mathcal{O} _{gH}|\neq |\mathcal{O} _{g^{-1}H}|$, then $0< |\mathcal{O} _{gH}|,|\mathcal{O} _{g^{-1}H}|\leq M$ for some $M>1$ independent of $g$.
-
@Mark For 1., what $g$ do you have in mind? Isn't your $|{\cal O}_{gH}|$ the number of elements of the form $x^mgx^n$ up to the equivalence $x^mgx^n \sim x^mgx^{n'}$? That looks infinite to me if the reduced word $g$ contains a $y$ or a $y^{-1}$, or else equal to 1. – David Feldman Apr 6 2012 at 3:53
I did not notice $|$ in 1. – Mark Sapir Apr 6 2012 at 7:49
Just a remark: you might do a search for the term "commensurator". en.wikipedia.org/wiki/Commensurator – Agol Apr 9 2012 at 22:54

I believe (1 and 2) and (3) are mutually exclusive. Here is a proof:

First, the commensurator $$Comm_G(H) = \{g\in G : |\mathcal{O} _{gH}|, |\mathcal{O} _{g^{-1}H}|<\infty\}$$ is a group. We will show:

Lemma: $\varphi\colon Comm_G(H)\to \mathbb{Q}_{>0}$ by $g\mapsto \displaystyle\frac{[H\colon H\cap gHg^{-1}]}{[H\colon H\cap g^{-1}Hg]}$ is a homomorphism.

From the lemma, if we assume there is a $g\in Comm_G(H)$ such that $\varphi(g)=x>1$ (criteria 1 and 2), then the order of $g$ must be infinite, since $x>1$ implies $x^n>1$ for all $n\geq 1$. Since the order of $g$ is infinite, criterion 3 cannot hold since eventually $\varphi(g^n)=\varphi(g)^n=x^n>M$ for any $M>0$.

Proof of the lemma:

We must show $\varphi(g_1g_2)=\varphi(g_1)\varphi(g_2)$. Define the following constants:

• For $i=1,2$, $a_i = [H\colon H\cap g_iHg_i^{-1}]$ and $b_i = [H\colon H\cap g_i^{-1}Hg_i]=[g_iHg_i^{-1}\colon H\cap g_iHg_i^{-1}]$
• $a=[H\colon H\cap (g_1g_2)H(g_1g_2)^{-1}]$ and $b=[(g_1g_2)H(g_1g_2)^{-1}\colon H\cap (g_1g_2)H(g_1g_2)^{-1}]$

Note that since $x\mapsto g_1xg_1^{-1}$ is an automorphism of $G$, we have:

• $a_2=[g_1Hg_1^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}]$ and $b_2=[(g_1g_2)H(g_1g_2)^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}]$

Now look at the subgroup $K=H\cap g_1 Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}$, and define

• $a_1'=[H\cap (g_1g_2)H(g_1g_2)^{-1}\colon K]$
• $a_2'=[H\cap g_1Hg_1^{-1}\colon K]$
• $b_1'=[g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}\colon K]$

which are all finite, since if we have a quadrilateral of groups $L_1\cap L_2\subset L_1,L_2\subset G$, we must have $[L_1\colon L_1\cap L_2] \leq [G\colon L_2]$. Now since index is multiplicative, we have

• $a a_1'=a_1a_2'$
• $ba_1' = b_2 b_1'$
• $a_2b_1'=b_1a_2'$

Solving for $a$ and $b$, we get $$\frac{a}{b} = \frac{a_1a_2'}{a_1'}\frac{a_1'}{b_2b_1'}=\frac{a_1a_2'}{b_2b_1'}.$$ Now note that $\displaystyle \frac{a_2'}{b_1'}=\frac{a_2}{b_1}$, so we have $$\varphi(g_1g_2)=\frac{a}{b} = \frac{a_1}{b_2}\frac{a_2}{b_1}= \frac{a_1}{b_1}\frac{a_2}{b_2}=\varphi(g_1)\varphi(g_2).$$

-
I think 1. and 3. are already incompatible by your argument. Also, just an aside: If $G$ is a totally disconnected, locally compact group and $H$ is an open compact subgroup, then the homomorphism you define is the modular function of the group: it measures the extent to which a right translation rescales a left Haar measure. – Colin Reid Apr 11 2012 at 7:52

The number of elements in ${\mathcal O}_{gH}$ is the index of $H\cap gHg^{-1}$ in $H$. Your condition 1 means that there is an element $g$ such that $|H:H\cap gHg^{-1}|\lt |H:H\cap g^{-1}Hg|=|gHg^{-1}: H\cap gHg^{-1}|$. So you want $H\cap gHg^{-1}$ to have different but finite indices in $H$ and in $gHg^{-1}$. The best way to achieve what you want (including 2 and 3) is to consider a group $H$ with two isomorphic subgroups $U,V$ of different finite indices, then consider the HNN extension $\langle H,g\mid gUg^{-1}=V\rangle$.

There are many such groups. For example, the Thompson group $F$ has a subgroup of index 2 that is isomorphic to itself (see this paper, Corollary 3.3, where all subgroups of finite index in $F$ which are isomorphic to $F$ are described.

Another source of examples is the lamplighter group $L=\mathbb{Z}_2\wr \mathbb{Z}$. If $a$ is the generators of $\mathbb{Z}_2$ and $b$ is the generator of $\mathbb{Z}$, then $L_k=\langle a, b^k\rangle$ is isomorphic to $L$ and has a finite index in $L$. The isomorphism from $L$ to $L_k$ is $a\mapsto a$, $b\mapsto b^k$ (here $k\ge 1$).

I think it is worthwhile to check whether properties 2 and 3 holds for these examples.

Warning I will leave the rest of the previous answer as an illustration of the fact that everybody should remember to use multiplication table properly. Thanks to Ian Agol and Dave Penneys for pointing out my error.

For example, take $H=F_2\times F_3$, $U=U_1\times U_2$ where $U_1$ is of index 10 in $F_2$, $U_2$ is of index 12 in $F_3$ (in that case, by Schreier's formula, the rank of $U_1$ is 11, the rank of $U_2$ is 25), $V=V_1\times V_2$ with $V_1$ of index 24, $V_2$ of index $5$ (then the rank of $V_1$ is 25 and the rank of $V_2$ is 11). In that case $U$ is isomorphic to $V$. The other properties should follow from the general properties of HNN extensions.

-
Thanks for your answer Mark! I've been working through the example, and 2 and 3 still seem non-trivial (especially 3). For example, $[H\colon H\cap t^2 H t^{-2}]$ seems highly dependent on how $t$ moves $U\cap V$ (and thus depends on the choice of isomorphism $U\cong V$). Are there any good references for the general properties of HNN extensions and why 2 and 3 should hold? Thanks again! – Dave Penneys Apr 9 2012 at 14:53
And aren't the indices the same as $[H\colon U]=(10)(12)=120=(24)(5)=[H\colon V]$? – Dave Penneys Apr 9 2012 at 18:12
I don't think this answer can possibly work. We have $[H:H\cap gHg^{-1}|= \chi(H\cap gHg^{-1})/\chi(H) = \chi(H \cap g^{-1}Hg)/\chi(H) = [H:H\cap g^{-1}Hg]$ (when the index is finite), so property (1) is not satisfied since $\chi(F_2\times F_3)=2$. Maybe there's some modification with groups with $\chi=0$. – Agol Apr 9 2012 at 18:33
Property 1 is not true in your example. $H$ must have euler characteristic $=0$ to have two isomorphic subgroups of different index. – Agol Apr 9 2012 at 22:13
I suppose the easiest example of this kind is $\mathbb{Z}$ with subgroups $n\mathbb{Z}$ and $m\mathbb{Z}$. The corresponding HNN extension is the Baumslag-Solitar group $BS(m,n)$. Property 1 holds when $n\not=m$, property 2 holds since $H=\IZ$ is an almost normal subgroup of $G=BS(m,n)$. But i am quite sure property 3 does not hold. (we should have something like $[H:H\cap t^k H t^{-k}]=(\frac{m}{n})^k$) A similar problem should happen in general with HNN extensions. – Steven Deprez Apr 10 2012 at 8:27