I have found in Wikipedia that the number of automorphisms of a Riemann surface of genus 6 does not exceed 150. The page offers neither a proof nor a reference. Can someone help me?
3 Answers
In case you don't know the general context: There is a curve of genus $g$ with endomorphism group contained in $G$ if and only if $G$ can be generated by elements $g_1$, $g_2$, ..., $g_k$ with orders $c_1$, $c_2$, ..., $c_k$, satisfying $g_1 g_2 \cdots g_k = 1$, obeying $$G = (g1) \left( \frac{1}{2} \sum (1c_i^{1})  1 \right)^{1}. $$ The largest possible value of the second factor occurs for $(c_1, c_2, c_3) = (2,3,7)$, in which case we get $G = 84 (g1)$. However, this bound is only achieved if there actually is a group of order $84 (g1)$ generated by elements of orders $2$, $3$ and $7$ with product $1$. For whatever reason^{1}, there is no such group of order $420$.
To get order $150$, we want $\sum (1c_i^{1}) = 2+\tfrac{1}{15}$; this occurs for $(2,3,10)$. I believe you can achieve this inside $S_3 \ltimes (\mathbb{Z}/5)^2$, taking elements in $S_3$ with product $1$ and orders $(2,3,2)$ and lifting them to the semidirect product so that one of the transpositions stays order $2$ and the other becomes order $10$.
If we want order $>150$ then we need $2 < \sum (1c_i^{1}) < 2+\tfrac{1}{15}$. Note that we must have three summands, since $(1/2)+(1/2)+(1/2)+(1/2)=2$ is too small and $(1/2)+(1/2)+(1/2) + (2/3) = 2+\tfrac{1}{6}$ is too large. So we can alternatively say that we want $\tfrac{14}{15} < c_1^{1}+c_2^{1}+c_3^{1} < 1$. Will Sawin and Roy Smith work out the options in the comments to another answer: You need to check $(2,3,7)$, $(2,3,8)$, $(2,3,9)$ and $(2,4,5)$, corresponding to groups of orders $420$, $240$, $180$ and $200$.
Footnote 1: According to the groupprops wiki, the only nonsolvable group of order $420$ is $A_5 \times \mathbb{Z}/7$ and thus every group of order $420$ has a nontrivial abelian quotient $A$. If you believe this reference, I can give a quick proof. If $A$ is abelian, then the only solutions to $g_1 g_2 g_3=1$ in $A$ with $g_1^2 = g_2^3=g_3^7=1$ are $g_1 =g_2=g_3=1$; prove this by considering $(g_1 g_2 g_3)^6$, $(g_1 g_2 g_3)^{14}$ and $(g_1 g_2 g_3)^{21}$. So any solution in $G$ lies in the kernel of $G \to A$, and thus can't generate.

5$\begingroup$ 150 is the order of the automorphism group of the Fermat quintic $x^5+y^5+z^5=0$. $\endgroup$– abxNov 29, 2017 at 17:15

2$\begingroup$ Ah, which is indeed $S_3 \ltimes (\mathbb{Z}/5)^2$: permute the variables and multiply them by roots of unity. $\endgroup$ Nov 29, 2017 at 17:53

3$\begingroup$ re: footnote 1. By the classification of finite simple groups of order dividing 420, if G of order 420 has no abelian quotient, it must have A_5 as a quotient. Then since (7,60)=1, it must be a semidirect product (Schur–Zassenhaus). Since A_5 can only act trivially on the cyclic group C_7, this must be a direct product. $\endgroup$ Nov 29, 2017 at 21:44

$\begingroup$ I have computed using MAGMA that there are no groups of order 240, 200 or 180 with generators with those orders. I would miss only a reference for the statement about the structure of the automorphism group, thank you. $\endgroup$– S.LiaJan 1, 2018 at 17:21
See "The locus of curves with prescribed automorphism group" by MagaardShaskaShpectorovVölklein, and the references therein.

1$\begingroup$ Could you say more precisely where the bound 150 for genus 6 curves is obtained? $\endgroup$– abxNov 29, 2017 at 16:54

$\begingroup$ Table 4 lists statistics for all automorphism groups of curves of genus $4 \leq g \leq 10$ with order $\geq 4(g1)$. I don't understand what all the table entries mean, but the group order is the first element of the ordered pair in the second column. $\endgroup$ Nov 29, 2017 at 16:57

1$\begingroup$ @abx. Maybe this is clear, but as I understand it, the maximum order of the group is either 84(g1), or 48(g1), or 36(g1) or 30(g1), ... according to whether the triple in David Speyer's answer can be (2,3,7), or (2,3,8), or (2,3,9) or (2,3,10),...Since these possibilities are 420, 240, 180, 150,...presumably you have to rule out the possibility of a curve of genus 6 having an automorphism (sub)group of order 4. ?? $\endgroup$ Nov 29, 2017 at 18:34

2$\begingroup$ @roysmith I believe you have missed $40(g1)$ from the triple $(2,4,5)$, but not any others. $\endgroup$ Nov 29, 2017 at 18:52

$\begingroup$ @roysmith Thank you! I was going to do this computation later today. Right, the computation is finite in theory; you just need to go through the groups of orders $420$, $240$, $200$ and $180$ and see if they can be generated as claimed. $\endgroup$ Nov 29, 2017 at 19:12
The question is purely combinatorial  the Hurwitz bound comes from the observation that the quotient of a surface by its automorphism group is a hyperbolic orbifold, and the hyperbolic orbifold of smallest area is the $(2, 3, 7)$ triangle orbifold, Dividing the area of your surface by the area of the orbifold gives you the Hurwitz bound. However, for some genera, there is no triangulation where the vertices have the right degrees, then you have to look for bigger triangle groups. For the surface of genus $6,$ you get the $(2, 3, 10)$ triangle group, but in any case, this is just a combinatorial exercise (for any fixed genus).

4$\begingroup$ Could you explain how you exclude (2,3,7), (2,3,8), (2,3,9) and (2,4,5) for genus 6 by combinatorial considerations? $\endgroup$– abxNov 30, 2017 at 8:17