# When are indiscrete reflection groups Coxeter groups?

A well-known theorem of Coxeter states that any discrete group $$W$$ which is generated by reflections across (possibly affine) hyperplanes in Euclidean space is a Coxeter group: it has a presentation with generators $$\{r_i, i \in I\}$$ and relations $$r_i^2=e, \forall i$$ and $$(r_ir_j)^{m_{ij}}=e$$ for some numbers $$m_{ij} \in \{2,3,\ldots\} \sqcup \{\infty\}$$.

I am interested in when the discreteness hypothesis can be discarded. For example, take the simplest indiscrete case: let $$W$$ be generated by reflections across two central hyperplanes meeting at a $$\pi$$-irrational angle. Then $$W$$ is not discrete since it contains a dense set of rotations, but $$W$$ is still a Coxeter group with generators $$\{r_1,r_2\}$$ corresponding to the two given reflections and $$m_{12}=\infty$$.

Note that I do not require $$W$$ to be generated by finitely many reflections, nor my Coxeter groups to have $$|I|$$ finite. However I would still be interested to know about any results particular to the finite case.

Questions:

1. Is any such group $$W$$ a Coxeter group? I suspect not.
2. As a particular example: is the orthogonal group a Coxeter group? Again, probably not, but I don't see how to prove this.
3. Is there some nice condition, more general than discreteness, which guarantees that $$W$$ is a Coxeter group?

This other question also asks for reflection groups that aren't Coxeter groups, but without focusing on discreteness. It also doesn't insist on working in Euclidean space and does assume finite $$|I|$$. It got no answers.

• @SamHopkins Is this different than my first sentence? Feb 10, 2021 at 14:06
• @SamHopkins that's right. Feb 10, 2021 at 14:11
• This paper discusses defining relations in classical groups: sciencedirect.com/science/article/pii/0021869378901321. It suggests that Sigrid Becken established the defining relations for the orthogonal group, but that paper is in German so I can't understand it. But "All relations between reflections are consequences of relations whose length is at most four." is interesting... Feb 10, 2021 at 14:18
• Even if I know a presentation for the group with the generating set of all reflections, it's still not obvious to me how I would decide if some subset of these form the simple generators for a Coxeter group presentation. Feb 10, 2021 at 14:57
• I will add details later on. The point is that $O(3)$ contains an isomorphic copy of a (uniform) irreducible higher rank lattice (a lattice in a product of several $SL(2,R)$'s) and those do not embed in Coxeter groups. Feb 10, 2021 at 16:57

Over on math.SE, I pointed out that there are three reflections $$a$$, $$b$$ and $$c$$ in $$O(3)$$ such that $$\langle a, b \rangle$$, $$\langle a, c \rangle$$ and $$\langle b, c \rangle$$ are infinite dihedral, but $$(abc)^6 = 1$$. However, I didn't actually prove that it wasn't a Coxeter group. Maybe this question will motivate someone to finish off the proof.