Here is a sketch of an argument that should prove that the set of curves of genus $g$ with automorphisms is closed.

As pointed out in the comments, by the Hurwitz bound there are at most finitely many groups that can act on a curve of genus $g$. Fix such a $G$ and fix a representation $V$ of $G$ of dimension $5g-5$. Again there are only finitely many such $V$'s up to isomorphism. Denote by $X_{V,G}$ the subset of curves in the moduli with a faithful $G$-action such that the representation of $G$ on $H^0(3K)$ is isomorphic to $V$. Denote by $H_{G,V}$ the subset of the Hilbert scheme of ${\mathbb P}(V)$ whose points correspond to the smooth curves $C$ of genus $g$ such that $C$ is $G$-invariant and the hyperplane section restricts to $3K_C$.
The set $H_{G,V}$ is quasi-projective variety and it maps onto $X_{G,V}$, so $X_{G,V}$ is constructible.

So, to prove that the set $X_G$ of curves with a faithful $G$-action is closed, it is enough to show that if a curve $C_0$ is the special fiber of a $1$-parameter family $\{C_t\}$ of smooth curves with a faithful $G$-action, then $C_0$ also has a faithful $G$-action. Up to base change we may assume that the base of the family is a small disk $D\subset {\mathbb C}$, hence that the total space $S$ of the family is a smooth surface. By assumption $G$ acts birationally on $S$. Hence it is known that it is possible to blow up $S$ and obtain a smooth surface $S'$ on which $G$ acts biregularly. The components of the special fiber $C_0'$ of $S'$ are the strict transform of $C_0$ and some rational curves. Since an automorphism cannot map a rational curve to a curve of positive genus, it follows that $G$ acts biregularly on $C_0$ (in fact by a similar argument one can show that $G$ acts biregularly on $S$).

The last thing to show is that the $G$-action on $C_0$ is faithful. So assume that $g\in G$, $g\ne 1$ fixes $C_0$ pointwise and let $x\in C$ be a general point. Then there are analytic coordinates $(z,t)$ near $x$ such that $C_0$ is given by $t=0$ and the $G$-action is given by $(z,t)\mapsto (z, ut)$, with $u\ne 1$ a root of unity. So locally $t=0$ is a section of the map $S\to D$ and $g$ acts nontrivially on this section. But we have a contradiction, since the action of $G$ preserves the fibers $C_t$ by construction.

Finally, the question whether $C/G$ is rational.
The genus of the quotient curve is equal to the number of times the trivial representation appears in the $G$-representation on $H^0(K_C)$.
EDIT: As above, let $f\colon S\to B$ be a smooth family of curves over a smooth connected curve $B$ such that $G$ acts on $S$ preserving the fibration. Then $f_*\omega_{S/B}$ is a rank $g$ vector bundle with a $G$-action and the fiber of $f_*\omega_{S/B}$ over $b\in B$ is $H^0(K_{C_b})$. The number of times the trivial representation appears in the decomposition of $H^0(K_{C_b})$ is the scalar product of the character of this representation with the trivial character. Hence it is a regular function of $b$ and, being an integer, it is constant. So the locus where $C/G$ is rational is closed, too.