In order to construct the coarse moduli scheme of smooth projective curves of genus $g$, the classical results of Mumford (using the numerical criterion of stability) say that for large enough $m$, the $m$th Hilbert point of a curve is stable. If I understand correctly, this implies that the Hilbert scheme $H_m$ of $m$canonical curves has a good quotient by the action of the relevant $PGL$ group, namely $PGL_{(2m1)(g1)}$. However, people usually deduce also that the Hilbert scheme $H_3$ of tricanonical curves has a good quotient by $PGL_{5(g1)}$ (this is the claim e.g. in the 1969 paper of Deligne and Mumford). I am probably missing something here: why is it so?

I do not believe the result about stability of tricanonical curves is a formal consequence of asymptotic stability. Rather, I think that the explicit arguments that prove $m$stability for $m\gg 0$, in fact, already apply MR0450272 (56 #8568) Reviewed Edit. I had misread Matthieu's question, which asks about smooth curves. In the paper above, Mumford proves $m$stability of "DeligneMumford stable curves" for $m\geq 5$. For the argument for smooth curves and $m\geq 3$, see the next edit. Second edit. Using the classical "Weierstrass Gap Theorem", in Section 4.6, Theorem 4.5, p. 95 of "Geometric Invariant Theory, 3rd ed.", Mumford proves Chow stability of every smooth curve of genus $g>1$ embedded in $\mathbb{P}^n$ by a complete linear system of degree $\geq 3g$ (also the open subscheme of Chow parameterizing smooth curves is canonically isomorphic to the corresponding open subscheme of the Hilbert scheme). So for $g\geq 3$, already bicanonically embedded curves are stable (in particular, the bicanonical morphism is an embedding). For $g=2$, the bicanonical morphism is not an embedding: it is the composition of the canonical map to $\mathbb{P}^1$ with a Veronese map $\mathbb{P}^1\to \mathbb{P}^2$. Thus, if you want to include $g=2$, you need the tricanonical embedding. So why doesn't Mumford say this explicitly in Chapter 5 of GIT? For one thing, at the time GIT was not sufficient to construct the coarse moduli space $M_g$ as a quasiprojective scheme over $\text{Spec}(\mathbb{Z})$, so the stability issue was a bit irrelevant (at that time). There were problems in positive characteristic (see the work of Haboush) and over $\text{Spec}(\mathbb{Z})$ (see the work of Seshadri). Happily, Mumford was able to avoid all of this by instead using some explicit invariant theory constructions that, essentially, avoid GIT altogether. This is the great irony of GIT: even though it is an indispensable tool in algebraic geometry, Mumford's first construction of $M_g$ really doesn't use GIT at all. 


In that DeligneMumford's paper they proved stability for m≥5*. To consider the pluricanonical embedding $\omega_C^m$ with $m=3$ gives a different compactification of $M_g$. In particular cusps are stable and elliptic tails are unstable. This case was worked out by Schubert (a Gieseker student). The article is: A new compactification of the moduli space of curves D Schubert  Compositio Math, 1991 http://archive.numdam.org/ARCHIVE/CM/CM_1991__78_3/CM_1991__78_3_297_0/CM_1991__78_3_297_0.pdf Finally, notice it is not obvious that the GIT quotient stabilizes (i.e it is the same for $m>>0$). This stabilization does not hold in some higher dimensional cases. 

