# Integral orthogonal group for indefinite ternary quadratic form

I have the indefinite quadratic form $q(x,y,z) = 19 x^2 + 5 y^2 - z^2.$ It's not my fault. I find, on reflection, that I have no idea how to describe the orthogonal group of this over the integers. The thing is isotropic in $\mathbb Z,$ for example $19 + 45 = 64,$ and $76 + 5 = 81.$ Even if that were not the case, the orthogonal group would contain those for the binary forms $19 x^2 - z^2$ and $5 y^2 - z^2.$ Note that there is an answer for the original Pythagorean problem, Orthogonal group of quadratic form and to some extent http://en.wikipedia.org/wiki/Tree_of_Pythagorean_triples Oh, note that i found all null vectors of the form, https://math.stackexchange.com/questions/483496/pythagorean-triples-with-additional-parameters/483510#483510 which took two recipes because of an annoying odd even thing.

Anyway, given the diagonal matrix $F$ with diagonal entries $19,5,-1,$ here are a number of integral matrices $P$ that solve $P^t F P = F,$ and so are called automorphs or members of the (automorphism, orthogonal, isometry, rotation) group. As we are in odd dimension, we do not much care about the determinant and allow $\pm 1.$

Note also that a recent Bulletin included two articles on the integral orthogonal group for an indefinite quaternary form, one by Kontorovich and one by Fuchs, anyway see http://www.ams.org/journals/bull/2013-50-02/

Um, let's see, oddity, you can take all entries of $P$ positive if you wish, so I'm just printing those. Since we can negate any row or any column at our leisure, it hurts nothing.

QUESTION: can someone please give generators for the entire group, that is every isometry can be written as a word in said generators? I'm hoping I've given enough information to do that. I'm guessing it should be no more than about five matrices, with any luck in this list or evident products of same.

EDIT: Reading Keith's eight-page note at http://www.math.uconn.edu/~kconrad/blurbs/ called Orthogonal Group of $x^2 + y^2 - z^2.$ Maybe I can do this myself, given time. Not sure. I did learn a bit about reflections for the "Euclidean Forms" project. Alright, Keith gives his five reflections at the bottom of page 2. So, let's see. need to collect a bunch of vectors where the quadratic form evaluates to one of $-2,-1,1,2.$ Do that tomorrow, easy enough.

List of some vectors with quadratic form equal to -2,-1,1,2 for 19 x^2 + 5 y^2 - z^2

                x           y           z
-----------------------------------------
-2          21          34         119
-2          21         170         391
-2          31          18         141
-2          39         110         299
-2          39         206         491
-2          59         186         489
-2          59         282         681
-2         131         270         831
-2         159         278         931
-2         201          94         901
-2         229          18         999
-2         231          70        1019
-----------------------------------------
-1           0           0          -1
-1           0           0           1
-1           0           4           9
-1           0          72         161
-1           1           1           5
-1           1           4          10
-1           1          11          25
-1           1          29          65
-1           1          76         170
-1           1         199         445
-1           4           8          25
-1           4          28          65
-1           4         172         385
-1          10          10          49
-1          10         106         241
-1          10         286         641
-1          14          10          65
-1          14         170         385
-1          15           4          66
-1          15          27          89
-1          15          39         109
-1          15          85         201
-1          15         113         261
-1          15         228         514
-1          15         300         674
-1          20           8          89
-1          20          76         191
-1          20          80         199
-1          20         284         641
-1          21          18         100
-1          21          23         105
-1          21          77         195
-1          21          87         215
-1          21         213         485
-1          21         238         540
-1          24           4         105
-1          24          84         215
-1          24         104         255
-1          25           1         109
-1          25          53         161
-1          25          56         166
-1          25         160         374
-1          25         167         389
-1          30          42         161
-1          30         266         609
-1          35          23         161
-1          35          46         184
-1          35         115         299
-1          35         161         391
-1          39           0         170
-1          39          85         255
-1          39         255         595
-1          45          15         199
-1          45          77         261
-1          45         122         336
-1          45         246         584
-1          49          11         215
-1          49          91         295
-1          49         124         350
-1          49         284         670
-1          50          46         241
-1          55          11         241
-1          55         104         334
-1          55         137         389
-1          71           8         310
-1          71         143         445
-1          71         167         485
-1          74          94         385
-1          79          28         350
-1          79          77         385
-1          79         133         455
-1          79         217         595
-1          85          53         389
-1          85         115         451
-1          85         274         716
-1          99          94         480
-1          99          99         485
-1         101          29         445
-1         101          91         485
-1         101         106         500
-1         101         179         595
-1         101         266         740
-1         105          77         489
-1         105         129         541
-1         111          15         485
-1         111         220         690
-1         111         265         765
-1         115          91         541
-1         115         134         584
-1         120         276         809
-1         125          94         584
-1         125         151         641
-1         129          87         595
-1         129         167         675
-1         130         134         641
-1         139          85         635
-1         139         190         740
-1         146         262         865
-1         151          56         670
-1         151         251         865
-1         154         182         785
-1         155          10         676
-1         155         106         716
-1         155         199         809
-1         171          77         765
-1         171         267         955
-1         174         186         865
-1         174         274         975
-1         179         133         835
-1         179         167         865
-1         179         182         880
-1         179         218         920
-1         185          29         809
-1         185          80         826
-1         185         293        1039
-1         190         218         961
-1         200         220        1001
-1         204         228        1025
-1         211          10         920
-1         211         115         955
-1         214         190        1025
-1         215          91         959
-1         220           4         959
-1         220          28         961
-1         221         143        1015
-1         221         293        1165
-1         224         124        1015
-1         225          57         989
-1         225          85         999
-1         230         122        1039
-1         245         137        1111
-1         249         104        1110
-1         251         151        1145
-1         251         179        1165
-1         275          77        1211
-1         276          76        1215
-1         295           8        1286
-1         295         167        1339
-----------------------------------------
1           0           1          -2
1           0           1           2
1           0          17          38
1          10           9          48
1          10         111         252
1          10         273         612
1          18           7          80
1          18         257         580
1          28          39         150
1          28         249         570
1          30          31         148
1          60           7         262
1          72         191         530
1          78           1         340
1          78         199         560
1          80         159         498
1          98         225         660
1         118         297         840
1         168         193         850
1         172           9         750
1         180         199         902
1         210         193        1012
1         210         263        1088
1         228         121        1030
1         270         191        1252
1         282          73        1240
1         298         105        1320
-----------------------------------------
2           3           0          13
2           3          52         117
2           7          16          47
2           7          44         103
2          13          16          67
2          13         124         283
2          17          28          97
2          17          32         103
2          17         124         287
2          17         136         313
2          27          16         123
2          27          92         237
2          27         120         293
2          37          28         173
2          37         104         283
2          37         196         467
2          43          76         253
2          47          32         217
2          57         180         473
2          57         272         657
2          87          24         383
2          87         180         553
2          93          60         427
2          93         196         597
2         147          24         643
2         163         280         947
2         173          52         763
2         173          76         773
2         193          44         847
2         197          92         883
2         197         260        1037
2         207          16         903
2         217          52         953
2         223          44         977
2         247         244        1207
2         277          52        1213
-----------------------------------------
x           y           z


LIST of some orthogonal matrices, P solves P^t F P = F:

  -1           0           0
0           1           0
0           0           1

1           0           0
0          -1           0
0           0           1

1           0           0
0           1           0
0           0          -1

1           0           0
0           1           0
0           0           1

1           0           0
0           9           4
0          20           9

1           0           0
0         161          72
0         360         161

39          10          10
38          11          10
190          50          49

39         110          50
38         101          46
190         530         241

39         290         130
38         299         134
190        1430         641

39        1990         890
38        1829         818
190        9590        4289

39          40          20
152         151          76
380         380         191

39          40          20
152         161          80
380         400         199

39          10          10
418         101         106
950         230         241

39         110          50
418        1211         550
950        2750        1249

39          10          10
1102         299         286
2470         670         641

170           0          39
0           1           0
741           0         170

170         780         351
0           9           4
741        3400        1530

210         140          79
76          49          28
931         620         350

210         320         151
76         119          56
931        1420         670

210          20          49
532          49         124
1501         140         350

210          20          49
1216         119         284
2869         280         670

360         250         139
494         341         190
1919        1330         740

360         530         251
494         731         346
1919        2830        1340

360         130         101
950         341         266
2641         950         740

550         480         249
228         201         104
2451        2140        1110

550         660         321
228         271         132
2451        2940        1430

550          60         129
1824         201         428
4731         520        1110

609         380         220
76          49          28
2660        1660         961

609         980         460
76         119          56
2660        4280        2009

609         320         200
1216         641         400
3800        2000        1249

609          20         140
1444          49         332
4180         140         961

759         730         370
646         619         314
3610        3470        1759

759         830         410
646         709         350
3610        3950        1951

759         220         200
836         241         220
3800        1100        1001

780         250         211
38          11          10
3401        1090         920

780        1970         899
38         101          46
3401        8590        3920

780          10         179
950          11         218
4009          50         920

780         670         349
1102         949         494
4199        3610        1880

780         950         461
1102        1339         650
4199        5110        2480

1500         310         371
494         101         122
6631        1370        1640

1500         130         349
1178         101         274
7049         610        1640

• I'm running out of steam. I had a dental crown first session today, and I'm wiped out and going to bed. Sorry if you stay up later than I do and need clarification of something, I'll need to attend to that tomorrow. Mostly I don't like it when people post a question and walk away, so I thought I would mention it in advance. Sep 5, 2013 at 5:18
• Wait, this looked like a good start but what's the question $-$ what do you want to know about this orthogonal group? Sep 5, 2013 at 5:19
• @Noam, hi, some description with a subset of these as generators, I would guess no more than about five. I experimented trying to write some of the matrices as products of the most attractive ones, but did not get very far. Thanks for your interest. I'll wait on your reply a bit. Sep 5, 2013 at 5:21
• @Noam, for example, the Apollonian group is generated by four matrices, each of which squares to the identity. At least I think they are saying the same thing. Sep 5, 2013 at 5:25
• Since the quadratic form represents a rational zero, the integral group is commensurable with $PSL(2,{\mathbb Z})\simeq SO(2,1)({\mathbb Z})$. It would be interesting to see what the congruence subgroup is, I suppose. Otherwise, the "general" answer is that is is a congruence subgroup of the integral modular group. Sep 6, 2013 at 10:23

Edit : this is a new answer, after more computations.

Let $H$ be the subgroup of your orthogonal group that preserve globally each connected component of the (two-sheeted) space $q(x,y,z)=-1$.

Up to this action, there is a single isometry class of isotropic vectors.

One representant is $(-1\ -3\ 8)$ and its stabilizer is the infinite dihedral group generated by

B1 = [ -39 -152  380]   C1 = [-2889 -8398 22610]
[ -40 -151  380]        [-2210 -6421 17290]
[ -20  -76  191]        [-1190 -3458  9311]


The group $H$ can be described as follows : let us write

A1 = [-1  0  0]             B1 = [ -39 -152  380]             B6=[ -609 -1216  3800]
[ 0  1  0]                  [ -40 -151  380]                [ -320  -641  2000
[ 0  0  1]                  [ -20  -76  191]                [ -200  -400  1249]

A2 = [ 1  0  0]             B2 = [-23751 -50350 152950]       T=[-609  -76 2660]
[ 0 -1  0]                  [-13250 -28091 85330]          [-380  -49 1660]
[ 0  0  1]                  [ -8050 -17066 51841]          [-220  -28  961]

A3 = [ 1  0  0]             B3 = [-1608009 -3459026 10438030]
[ 0 -9 20]                  [ -910270 -1958101  5908810]
[ 0 -4  9]                  [ -549370 -1181762  3566111]

A4 = [-170    0  741]           B4 = [-194561 -415872 1258560]
[   0    1    0]                [-109440 -233929  707940]
[ -39    0  170]                [ -66240 -141588  428489]

A5 = [ -930 -2128  6251]        B5 = [-39 -38 190]
[ -560 -1279  3760]             [-10 -11  50]
[ -329  -752  2210]             [-10 -10  49]


All the matrices $Ai$ and $Bi$ have order 2, while $T$ has infinite order.

Let $K$ be the free product of all the subgroups generated by these element (thus it is isomorphic to $\mathbf Z/2^{\star 11}\star \mathbf Z$.

Let $R$ be the subgroup of K generated by $[A1,A2]$ , $[A1,A3]$, $[A3,B1]$, $[A2,A4]$, $[A5,B3]$, and $T.A4.T^{-1}.A5$.

Then $R$ is the kernel of the obvious representation of $K$ in $H$, which is an epimorphism.

The group $\mathrm H_1(H,\mathbf Q)$ has dimension 1, thus $H$ is not a reflective group. I hope to be able to draw a Coxeter Diagram for its reflection subgroup soon. If this happens, I will edit this answer once more.

• The method uses Voronoi cells ... Note that this group is non-reflective ... and I believe (but might be wrong) that Vinberg algorithm should not finish. If I had an easy way to plot hyperbolic triangles, I could draw the fundamental polygon, but I just tried with Sage and it did not work ... Sep 5, 2013 at 14:58
• Thanks. So, this group is not generated by reflections? Sep 5, 2013 at 19:15
• It cannot : its rational homology would be trivial, and its not the case. Sep 5, 2013 at 19:26
• @Sasha : excuse my ignorance, but I am pretty sure to have heard once that it may happen that the algorithm terminates, but the polyhedron obtained has infinite volume ... and hence cannot be of finite index in the automorphism group of the quadratic form ... Would you have a reference ? Sep 5, 2013 at 21:14
• @Sasha : I still don't have enough reps to comment your answer. I seem to find that the Apollonian group is a reflection group (with 6 generators). Sep 5, 2013 at 22:02

The following paper could become of your interest:

"On groups of unit elements of certain quadratic forms", by È. B. Vinberg
http://mr.crossref.org/iPage?doi=10.1070%2FSM1972v016n01ABEH001346

Abstract: It is shown that, for the groups of unit elements of certain integral quadratic forms of signature $(n,1)$, there exist subgroups of finite index, generated by reflections; and the generators and relations of these subgroups are found.

Recently, quadratic forms $-3x_0^2 + x_1^2 + ... + x_n^2$ have been studied by John McLeod (http://arxiv.org/abs/1007.2299). Up to $n=13$ one can describe the group of units (i.e. automorphisms of this quadratic form) by using Vinberg's algorithm.

Hope this may help in some way. Cheers!

A little wild speculation: I don't know much about the Apollonian group, but seemingly it could be a Coxeter-like group, since each generator squares to the identity (is a "reflection") and there are four of them, corresponding to the four facets of a simplex ...

• Well, my little wild speculation is really too wild there: the groups is non-reflective! Again. Sep 5, 2013 at 16:22
• Thanks for trying to explain. What does it mean for a group to be "non-reflective?" I'm confused; as you say, the Apollonian group is evidently generated by four reflections...Also, it appears the group in my problem above cannot even be generated by reflections? Sep 5, 2013 at 19:13
• In case of interest, the Apollonian group shares the main phenomenon in the Markov equation $x^2 + y^2 + z^2 = 3 x y z,$ in that the sum of the squares is equal to a symmetric polynomial with each term "squarefree," so movements are accomplished by "Vieta Jumping" ; Apollonian is $$w^2 + x^2 + y^2 + z^2 = 2 w x + 2 w y + 2 w z + 2 x y + 2 x z + 2 y z$$ Sep 5, 2013 at 19:24
• I'm sorry, I messed up everything: I think that here "reflective" means that the maximal subgroup of $O(q)$ generated by reflections is finite-index in $O(q)$. Indeed, the group $O(q)$, with $q$ as in your question, is generated by 10 reflections and 2 isometries of infinite order, and is not reflective (because of the homology argument above), according to few_reps. The Apollonian group is reflective but has infinite co-volume. Sep 5, 2013 at 21:07
• Sasha, thank you. The general reason I am interested in isometry groups for indefinite integral forms is a very long story that culminated in this: mathoverflow.net/questions/69444/… The information in my self-answer is known to a very, very few, such as Richard Borcherds. Sep 5, 2013 at 21:37

For quadratic forms of signature $$(1,n)$$, we can use the theory of perfect forms following the book AMRT. It is relatively expensive to compute but it allows you to resolve a series of questions effectively. Note that it is more general than the Coxeter group approach of Vinberg, which requires the lattice to be reflective for the method to work.

The idea is to compute the perfect forms by the classic algorithm of Voronoi adapted to the hyperbolic case. A side product of this is a set of generating elements of the automorphism group of the quadratic form.

Also, the isotropic vectors are extreme rays of the perfect cones, thus from the classification of the perfect cones, we can derive a classification of the isotropic vectors.

I am 8 years after the initial message. Let me know if effective computations are welcomed.

• Thank you. I don't think I know a book AMRT, at least not by those initials. Nov 3, 2021 at 23:26
• It is Ash, Mumford, Rapoport and Tai, "Smooth compactifications of locally symmetric varieties" Nov 4, 2021 at 14:17
• Thanks again. In case of interest, see brief answer about computations mathoverflow.net/questions/389721/… Nov 4, 2021 at 16:44