$M_5$ is pretty big. I'll do $M_3$, which should get across the same points.
In $M_3$, there are two strata:
The hyperelliptic locus, $H$, consists of those curves which have a degree $2$ map to $\mathbb{P}^1$.
The planar biquadrics, $B$, consist of curves which can be embedded in $\mathbb{P}^2$ with degree $4$. Such a curve has many different degree $3$ maps to $\mathbb{P}^1$; projection from any of its points gives such a map.
Every curve in $M_3$ is in precisely one of $B$ and $H$.
There are several points I want to make:
(a) The topology on $M_3$ is not the disjoint union of $B$ and $H$. Rather, $B$ is dense and open in $M_3$, and $H$ is a hypersurface.
(b) There is no notion of "the defining equation" of a curve. The curves in $B$ can be embedded in $\mathbb{P}^2$ by degree $4$ equations; but they can also be embedded in $\mathbb{P}^3$ with various larger degrees and they can be embedded in the plane with higher degrees if you allow nodes.
The issue is more dramatic for curves in $B$. They can be embedded in $\mathbb{P}^1 \times \mathbb{P}^1$, with degree $(2,4)$. To embed them in $\mathbb{P}^2$, you have to allow nodes (or worse singularities); I'm not sure what the lowest degree you can get away with is there.
(c) It is interesting to see how a curve in $B$ degenerates to a hyperelliptic curve. Let $F_t$ be a family of degree $4$ curves in $\mathbb{P}^2$, parameterized by $t$ in a disc $\Delta$. Let $C \to \Delta \setminus \{ 0 \}$ be the corresponding family of abstract curves, and suppose the limit in $M_3$ as $t \to 0$ is a hyperelliptic curve. If we normalize things properly, we can take $F_0 = Q^2$ for a conic $Q$. Let $F_t = Q^2 + t G + \cdots$.
Let $\Delta'$ be the branched cover $t=u^2$ of $\Delta$. Then we can complete the family $C' \to \Delta' \setminus \{ 0 \}$ to a family $\tilde{C} \to \Delta'$ whose fiber over $0$ is the hyperelliptic curve in question.
The family $\tilde{C}$ maps to $\mathbb{P}^2$. For $u \neq 0$, the curve $C'_u$ maps to $\{ F_{u^2}=0 \}$. At $u=0$, the hyperelliptic curve double covers $Q$, ramified over the $8$ points $\{ Q=G=0 \}$.
(d) Similarly, we can take a family of genus $3$ curves with degree $3$ maps to $\mathbb{P}^1$ and take the limit of that family, in the sense of stable maps. I'm finding this limit a bit difficult. I think you get a curve with two components, $X \cup Y$, glued along a single node; where $X$ is hyperelliptic of genus $3$ and double covers $\mathbb{P}^1$ while $Y$ is genus $0$ and maps isomorphically to $\mathbb{P}^1$.
The case of genus 5 will be similar. There will be a generic situation, which in this case is degree $8$ curves in $\mathbb{P}^4$. (And those curves do, indeed, have degree $4$ maps to $\mathbb{P}^1$.) The other strata will be contained in the closure of it.
There is no one notion of "the defining equation" or "the degree" of an abstract curve. However, given a particular family of curves mapping to $\mathbb{P}^k$, we can take the limit in the sense of stable maps and get a stable map, one of whose components will be the limiting abstract curve.
