In order to construct the coarse moduli scheme of smooth projective curves of genus $g$, the classical results of Mumford (using the numerical criterion of stability) say that for large enough $m$, the $m$th Hilbert point of a curve is stable. If I understand correctly, this implies that the Hilbert scheme $H_m$ of $m$canonical curves has a good quotient by the action of the relevant $PGL$ group, namely $PGL_{(2m1)(g1)}$. However, people usually deduce also that the Hilbert scheme $H_3$ of tricanonical curves has a good quotient by $PGL_{5(g1)}$ (this is the claim e.g. in the 1969 paper of Deligne and Mumford). I am probably missing something here: why is it so?
I do not believe the result about stability of tricanonical curves is a formal consequence of asymptotic stability. Rather, I think that the explicit arguments that prove $m$stability for $m\gg 0$, in fact, already apply if m ≥ 3 if $m\geq 5$. The standard reference is the following.
MR0450272 (56 #8568) Reviewed
Mumford, David
Stability of projective varieties.
Enseignement Math. (2) 23 (1977), no. 12, 39–110.
14D20
Edit. I had misread Matthieu's question, which asks about smooth curves. In the paper above, Mumford proves $m$stability of "DeligneMumford stable curves" for $m\geq 5$. For the argument for smooth curves and $m\geq 3$, see the next edit.
Second edit. Using the classical "Weierstrass Gap Theorem", in Section 4.6, Theorem 4.5, p. 95 of "Geometric Invariant Theory, 3rd ed.", Mumford proves Chow stability of every smooth curve of genus $g>1$ embedded in $\mathbb{P}^n$ by a complete linear system of degree $\geq 3g$ (also the open subscheme of Chow parameterizing smooth curves is canonically isomorphic to the corresponding open subscheme of the Hilbert scheme). So for $g\geq 3$, already bicanonically embedded curves are stable (in particular, the bicanonical morphism is an embedding). For $g=2$, the bicanonical morphism is not an embedding: it is the composition of the canonical map to $\mathbb{P}^1$ with a Veronese map $\mathbb{P}^1\to \mathbb{P}^2$. Thus, if you want to include $g=2$, you need the tricanonical embedding.
So why doesn't Mumford say this explicitly in Chapter 5 of GIT? For one thing, at the time GIT was not sufficient to construct the coarse moduli space $M_g$ as a quasiprojective scheme over $\text{Spec}(\mathbb{Z})$, so the stability issue was a bit irrelevant (at that time). There were problems in positive characteristic (see the work of Haboush) and over $\text{Spec}(\mathbb{Z})$ (see the work of Seshadri). Happily, Mumford was able to avoid all of this by instead using some explicit invariant theory constructions that, essentially, avoid GIT altogether. This is the great irony of GIT: even though it is an indispensable tool in algebraic geometry, Mumford's first construction of $M_g$ really doesn't use GIT at all.

$\begingroup$ Well, but firstly Deligne and Mumford published their paper in 1969 (8 years before [SPV]:=Stability of proj. var.) and secondly, when you read [SPV], you see plenty of arguments that really work only for $m\gg 0$! $\endgroup$ – Matthieu Romagny Dec 2 '13 at 8:33

$\begingroup$ ps: quote from Deligne and Mumford: "By the results of [M1] (=Mumford's GIT book), we know that a coarse geometric quotient $M_g^0=H_g^0/PGL(5g6)$ exists (...)". $\endgroup$ – Matthieu Romagny Dec 2 '13 at 8:36

$\begingroup$ Wait, are you looking only at the parameter space for smooth curves? $\endgroup$ – Jason Starr Dec 2 '13 at 12:17


1$\begingroup$ Dear Jason, this discussion was very useful. I suggest that you post an answer to tell about the two constructions in Mumford's book, telling that the first does not prove that $M_g$ is a quotient of the tricanonical Hilbert scheme but that the second one (using moduli of abelian varieties) does prove it. I'll accept this answer. $\endgroup$ – Matthieu Romagny Dec 2 '13 at 15:49
In that DeligneMumford's paper they proved stability for m≥5*. To consider the pluricanonical embedding $\omega_C^m$ with $m=3$ gives a different compactification of $M_g$. In particular cusps are stable and elliptic tails are unstable. This case was worked out by Schubert (a Gieseker student). The article is: A new compactification of the moduli space of curves D Schubert  Compositio Math, 1991
http://archive.numdam.org/ARCHIVE/CM/CM_1991__78_3/CM_1991__78_3_297_0/CM_1991__78_3_297_0.pdf
Finally, notice it is not obvious that the GIT quotient stabilizes (i.e it is the same for $m>>0$). This stabilization does not hold in some higher dimensional cases.

$\begingroup$ Yes, I also noted in my earlier edit that the result is for $m\geq 5$. $\endgroup$ – Jason Starr Dec 2 '13 at 15:30

$\begingroup$ Does not quite answer my question, but that's interesting, thanks. $\endgroup$ – Matthieu Romagny Dec 2 '13 at 15:50

$\begingroup$ dear Matthieu, I am bit confused! You asked "people usually deduce also that the Hilbert scheme H3 of tricanonical curves has a good quotient by PGL5(g−1)..why is it so?". I have the impression that the Schubert's article constructs that GIT quotient. $\endgroup$ – eventually Dec 2 '13 at 17:20

$\begingroup$ Dear pmath, let me just emphasize that my question was about the way one deduces (if possible, which apparently is not the case) 3stability from $m$stability. Also, my question was only about smooth curves and had nothing to do with compactifications. However, your answer is certainly interesting and related to other aspects of GIT stability that are implicit in my question. $\endgroup$ – Matthieu Romagny Dec 2 '13 at 20:44

$\begingroup$ You are right, it is my mistake. However, I think that Theorem 5.1 and Lemma 5.2 of Schubert's article answer your question. $\endgroup$ – eventually Dec 2 '13 at 21:37