Let $C$ be a curve of genus $g \geq 2$. Then for $m_1 \geq 5$ the line bundle $m_1K_C$ induces an embedding in some $\mathbb{P}^{n}$ where $n:=n(m_1,g)$ depends of the genus of the curve and of $m_1$. As a consequence, we can see all our curves of genus $g$ as subschemes in a fixed projective space. This is useful, because we know there is a fine moduli space associated those subschemes! It is called: the Hilbert scheme. Of course, this space parametrizes other schemes, not only curves [1]. Let me denote as $Hilb_{C \hookrightarrow P^{n(m,g)}}$ the locally closed locus in the Hilbert scheme that parametrizes smooth curves of genus $g$ [2].
We want to take the GIT quotient
$$
\big( Hilb_{C \hookrightarrow P^{n}} \big)^{ss}//Aut(\mathbb{P}^n)
$$
To take this quotient, we want (need?) to have the Hilbert scheme inside a projective variety. This is a classical construction due to Grothendieck. We consider the exact sequence
$$
0 \to \mathcal{I}_C \to \mathcal{O}_{P^{n}}
\to \mathcal{O}_C \to 0
$$
which induces:
$$
0 \to H^0(C,\mathcal{I}_C(k)) \to
H^0(P^{n}, \mathcal{O}_{P^{n}}(k))
\to H^0(C, \mathcal{O}_C(k))
\to
H^1(C,\mathcal{I}_C(k))
\to 0
$$
for $m$ large enough (this depends only of the Hilbert polynomial) the groups
$H^1(C,\mathcal{I}_C(k))$ vanish for all $k \geq m$ ([3]). In particular, we can parametrize each subspace $H^0(C,I_C(m))$ with a point in a Grassmanian $Gr(r_1,r_2)$ where $r_1$ and $r_2$ depend of $m$. Those points are called the $m$-Hilbert points. Moreover, for any $m_1$, if we take $m$ large enough, then our GIT quotient is $\overline{M}_g$. If you use lower values of $m$, then you may obtain other birational models of $\overline{M}_g$.
An example of an important question related to this topic:
For a given $m_1$ and a low value of $m$, Is the map
$$
Hilb_{(C \hookrightarrow P^{n})}\hookrightarrow Gr(r_1,r_2)
$$
an embedding ?
Few things to care about:
[1] The Hilbert scheme parametrizes subschemes with a given constant Hilbert polynomial. In general, they can be horribly singular (both the objects and the Hilbert scheme itself)
[2] We want to carve out a "good loci" in the Hilbert scheme. It is not obvious this loci is either locally closed, open or even supported in a scheme. For curves, you can read Mumford's book on GIT (Proposition 5.1, and 5.3. I am using the third edition). For other moduli problems, please look at Kovac's notes "YPG to moduli of higher dimensional varieties" Section 5.C
[3] There is more to say, but I cannot recall all details or the geometric interpretation...
