Since you asked for details worked out, here are some notes from my course, with some details, but not so many examples. Since deformations occur by deforming the quotient curve and the branch points, as you said the rigid examples have quotient curve of genus zero, and three branch points. These occur below as realizing 84(g-1) and 48(g-1). These calculations also show that if there are 4 or more branch points on the genus zero quotient, hence at least a one parameter deformation, then the bound is ≤ 12(g-1). If there are either ≥ 5 branch points or the quotient has positive genus, there are ≤ 4(g-1) automorphisms.
Now suppose there are k points p1,....,pk of X/G which are branch points of the quotient map π:X--->X/G for the action of G on X. Then there exist k conjugacy classes of subgroups of G, containing groups of orders rj, j=1,...,k, such that over each point pj, there are |G|/rj ramified points for π, each with ramification index (rj-1). Thus the total ramification index of the map π is ∑ (|G|/rj)(rj-1). Hence the Hurwitz formula for the Euler characteristic reads chitop(X) = |G|chitop(X/G) - ∑ (|G|/rj)(rj-1) =
|G|[chitop(X/G)- ∑ (1-1/rj), summed from j=1,...k.
Equivalently, 2g(X) - 2 = |G|[2g(X/G) – 2 + ∑ (1-1/rj)]. (*)
This allows one to compute bounds on the size of |G|. Riemann surfaces of genus g(X) ≤ 1 can have arbitrarily large finite groups of automorphisms, e.g. any finite cyclic group can occur, but this is not true in higher genus. When g≥ 2 Hurwitz gave the general bound of |G| ≤ 84(g-1) as we discuss next.
Assume g(X) ≥ 2. If g(X/G) ≥ 1 and there is no ramification, then ∑ (1-1/rj)] = 0, so 2 ≤ 2g(X) -2 = |G|[2g(X/G) – 2], so the RHS is positive, whence 2g(X/G) - 2 ≥ 2, and thus 2g(X) -2 ≥ 2 |G|, so |G| ≤ (g-1).
If ∑ (1-1/rj)] > 0, then rj ≥ 2 implies ∑ (1-1/rj)] ≥ ½. Then g(X/G) ≥ 1 implies RHS(*) ≥(1/2) |G|, so |G| ≤ 4g(X)-4 = 4(g(X)-1).
If g(X/G) = 0, then g(X) ≥ 2 implies again RHS(*) > 0, so ∑ (1-1/rj)] > 2.
Now if ∑ (1-1/rj) = (1 – 1/r1) + (......)+ (1 – 1/rk) > 2, then certainly k > 2, since there are k terms added, each smaller than 1.
k ≥ 5: In all cases, ∑ (1-1/rj) = k - ∑ 1/rj ≥ k/2, so if k ≥ 5, then ∑ (1-1/rj) ≥ 2 + (1/2), and
(*) becomes 2g(X) - 2 ≥ |G|[1/2], so |G| ≤ 4(g-1).
k = 4: Then ∑ (1-1/rj) = 4 – 1/r1 – 1/r2 – 1/r3 – 1/r4. For this to be > 2, some rj must be > 2. Thus ∑ (1-1/rj) ≥ 4 – ½ - ½ - ½ - 1/3 = 2 + (1/6), whence |G| ≤ 12(g-1).
k = 3: If all rj ≥ 4, then ∑ (1-1/rj) ≥ 2 + (1/4), whence |G| ≤ 8(g-1).
If some rj = 3, no rj = 2, then ∑ (1-1/rj) ≥ 3 – 1/3 – 1/3 – ¼ = 2 + (1/12), so |G| ≤ 24(g-1).
If some rj = 2, no rj = 3, then ∑ (1-1/rj) ≥ 3 – ½ - ¼ - 1/5 = 2 + (1/20), so |G| ≤ 40(g-1).
r1 = 2, r2 = 3, then r3 ≥ 7, so ∑ (1-1/rj) ≥ 2 + (1/42), so |G| ≤ 84(g-1).
This is the largest possible bound when g(X) ≥ 2. I.e. g(X) ≥ 2 implies no finite group of order > 84(g-1) can act as automorphisms of X. If this bound does not occur, then the next best bound is r1 = 2, r2 = 3, then r3 = 8, so ∑ (1-1/rj) = 2 + (1/24), so |G| ≤ 48(g-1).
Hence, if for example, X does not have an automorphism of order 7, then the largest finite subgroup of Aut(X) has order ≤ 48(g-1).
Exercise: The genus three “Klein” curve x^3y + y^3z + z^3x = 0, has #(Aut(X)) = 168 = 84(g-1). Indeed Aut(X) = PSL(2,Z/7).
Exercise: If g(X) = 5, then no automorphism of X can have order 7, hence #(Aut(X)) = 84(g-1) cannot occur, but there is a curve X of genus 5 with 192 = 48(g-1) automorphisms. Indeed Aut(X) = PSL(2,Z/8), where Z/8 is the ring of integers mod 8. (See Gunning's Lectures on modular forms, p.15.)
Fact: The next genus in which a group of automorphisms occurs achieving Hurwitz maximal bound is g = 7, where there is a curve X with 504 automorphisms and Aut(X) = PSL(2,F8), where F8 is the field with 8 elements. This example was found by Fricke in 1899, forgotten and rediscovered by Macbeath in 1965. Macbeath also showed that curves with the maximal number 84(g-1) of automorphisms occur for infinitely many values of g.
