First, the commensurator $$Comm_H(GComm_G(H) = \{g\in G : |\mathcal{O} _{gH}|, |\mathcal{O} _{g^{-1}H}|<\infty\}$$ is a group. We will show:
Lemma: $\varphi\colon Comm_H(G)\to Comm_G(H)\to \mathbb{Q}_{>0}$ by $g\mapsto \displaystyle\frac{[H\colon H\cap gHg^{-1}]}{[H\colon H\cap g^{-1}Hg]}$ is a homomorphism.
From the lemma, if we assume there is a $g\in Comm_H(G)$ Comm_G(H)$such that$\varphi(g)=x>1$(criteria 1 and 2), then the order of$g$must be infinite, since$x>1$implies$x^n>1$for all$n\geq 1$. Since the order of$g$is infinite, criterion 3 cannot hold since eventually$\varphi(g^n)=\varphi(g)^n=x^n>M$for any$M>0$. Proof of the lemma: We must show$\varphi(g_1g_2)=\varphi(g_1)\varphi(g_2)$. Define the following constants: • For$i=1,2$,$a_i = [H\colon H\cap g_iHg_i^{-1}]$and$b_i = [H\colon H\cap g_i^{-1}Hg_i]=[g_iHg_i^{-1}\colon H\cap g_iHg_i^{-1}]$•$a=[H\colon H\cap (g_1g_2)H(g_1g_2)^{-1}]$and$b=[(g_1g_2)H(g_1g_2)^{-1}\colon H\cap (g_1g_2)H(g_1g_2)^{-1}]$Note that since$x\mapsto g_1xg_1^{-1}$is an automorphism of$G$, we have: •$a_2=[g_1Hg_1^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}]$and$b_2=[(g_1g_2)H(g_1g_2)^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}]$Now look at the subgroup$K=H\cap g_1 Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}$, and define •$a_1'=[H\cap (g_1g_2)H(g_1g_2)^{-1}\colon K]$•$a_2'=[H\cap g_1Hg_1^{-1}\colon K]$•$b_1'=[g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}\colon K]$which are all finite, since if we have a quadrilateral of groups$L_1\cap L_2\subset L_1,L_2\subset G$, we must have$[L_1\colon L_1\cap L_2] \leq [G\colon L_2]$. Now since index is multiplicative, we have •$a a_1'=a_1a_2'$•$ba_1' = b_2 b_1'$•$a_2b_1'=b_1a_2'$Solving for$a$and$b$, we get $$\frac{a}{b} = \frac{a_1a_2'}{a_1'}\frac{a_1'}{b_2b_1'}=\frac{a_1a_2'}{b_2b_1'}.$$ Now note that$\displaystyle \frac{a_2'}{b_1'}=\frac{a_2}{b_1}$, so we have $$\varphi(g_1g_2)=\frac{a}{b} = \frac{a_1}{b_2}\frac{a_2}{b_1}= \frac{a_1}{b_1}\frac{a_2}{b_2}=\varphi(g_1)\varphi(g_2).$$ 2 some typos fixed I believe (1) 1 and 2) and (3) are mutually exclusive. Here is a proof: First, the commensurator$Comm_H(G) Comm_H(G) = \{g\in G : |\mathcal{O} _{gH}|, |\mathcal{O} _{g^{-1}H}|<\infty\} $$is a group. We will show: Lemma: \varphi\colon Comm_H(G)\to \mathbb{Q}_{>0} by g\mapsto \displaystyle\frac{[H\colon H\cap gHg^{-1}]}{[H\colon H\cap g^{-1}Hg]} is a homomorphism. From the lemma, if we assume there is a g\in G Comm_H(G) such that \varphi(g)=x>1 (criterion 1)criteria 1 and 2), then the order of g must be infinite, since x>1 implies x^n>1 for all n\geq 1. Since the order of g is infinite, criterion 3 cannot hold since eventually \varphi(g^n)=\varphi(g)^n=x^n>M for any M>0. Proof of the lemma: We must show \varphi(g_1g_2)=\varphi(g_1)\varphi(g_2). Define the following constants: • For i=1,2, a_i = [H\colon H\cap g_iHg_i^{-1}] and b_i = [H\colon H\cap g_i^{-1}Hg_i]=[g_iHg_i^{-1}\colon H\cap g_iHg_i^{-1}] • a=[H\colon H\cap (g_1g_2)H(g_1g_2)^{-1}] and b=[(g_1g_2)H(g_1g_2)^{-1}\colon H\cap (g_1g_2)H(g_1g_2)^{-1}] Note that since x\mapsto g_1xg_1^{-1} is an automorphism of G, we have: • a_2=[g_1Hg_1^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}] and b_2=[(g_1g_2)H(g_1g_2)^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}] Now look at the subgroup K=H\cap g_1 Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}, and define • a_1'=[H\cap (g_1g_2)H(g_1g_2)^{-1}\colon K] • a_2'=[H\cap g_1Hg_1^{-1}\colon K] • b_1'=[g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}\colon K] which are all finite, since if we have a quadrilateral of groups L_1\cap L_2\subset L_1,L_2\subset G, we must have [L_1\colon L_1\cap L_2] \leq [G\colon L_2]. Now since index is multiplicative, we have • a a_1'=a_1a_2' • ba_1' = b_2 b_1' • a_2b_1'=b_1a_2' Solving for a and b, we get$$ \frac{a}{b} = \frac{a_1a_2'}{a_1'}\frac{a_1'}{b_2b_1'}=\frac{a_1a_2'}{b_2b_1'}. $$Now note that \displaystyle \frac{a_2'}{b_1'}=\frac{a_2}{b_1}, so we have$$ \varphi(g_1g_2)=\frac{a}{b} = \frac{a_1}{b_2}\frac{a_2}{b_1}= \frac{a_1}{b_1}\frac{a_2}{b_2}=\varphi(g_1)\varphi(g_2). $$1 I believe (1) and (3) are mutually exclusive. Here is a proof: First, the commensurator Comm_H(G) is a group. We will show: Lemma: \varphi\colon Comm_H(G)\to \mathbb{Q}_{>0} by g\mapsto \displaystyle\frac{[H\colon H\cap gHg^{-1}]}{[H\colon H\cap g^{-1}Hg]} is a homomorphism. From the lemma, if we assume there is a g\in G such that \varphi(g)=x>1 (criterion 1), then the order of g must be infinite, since x>1 implies x^n>1 for all n\geq 1. Since the order of g is infinite, criterion 3 cannot hold since eventually \varphi(g^n)=\varphi(g)^n=x^n>M for any M>0. Proof of the lemma: We must show \varphi(g_1g_2)=\varphi(g_1)\varphi(g_2). Define the following constants: • For i=1,2, a_i = [H\colon H\cap g_iHg_i^{-1}] and b_i = [H\colon H\cap g_i^{-1}Hg_i]=[g_iHg_i^{-1}\colon H\cap g_iHg_i^{-1}] • a=[H\colon H\cap (g_1g_2)H(g_1g_2)^{-1}] and b=[(g_1g_2)H(g_1g_2)^{-1}\colon H\cap (g_1g_2)H(g_1g_2)^{-1}] Note that since x\mapsto g_1xg_1^{-1} is an automorphism of G, we have: • a_2=[g_1Hg_1^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}] and b_2=[(g_1g_2)H(g_1g_2)^{-1}\colon g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}] Now look at the subgroup K=H\cap g_1 Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}, and define • a_1'=[H\cap (g_1g_2)H(g_1g_2)^{-1}\colon K] • a_2'=[H\cap g_1Hg_1^{-1}\colon K] • b_1'=[g_1Hg_1^{-1}\cap (g_1g_2)H(g_1g_2)^{-1}\colon K] which are all finite, since if we have a quadrilateral of groups L_1\cap L_2\subset L_1,L_2\subset G, we must have [L_1\colon L_1\cap L_2] \leq [G\colon L_2]. Now since index is multiplicative, we have • a a_1'=a_1a_2' • ba_1' = b_2 b_1' • a_2b_1'=b_1a_2' Solving for a and b, we get$$ \frac{a}{b} = \frac{a_1a_2'}{a_1'}\frac{a_1'}{b_2b_1'}=\frac{a_1a_2'}{b_2b_1'}. $$Now note that \displaystyle \frac{a_2'}{b_1'}=\frac{a_2}{b_1}, so we have$$ \varphi(g_1g_2)=\frac{a}{b} = \frac{a_1}{b_2}\frac{a_2}{b_1}= \frac{a_1}{b_1}\frac{a_2}{b_2}=\varphi(g_1)\varphi(g_2).