As already said by Simon in his comment, this is a very vast topic.

Let us stick for simplicity to the case of smooth surfaces $S$ of general type. 

The existence of a quasiprojective coarse moduli space $\mathfrak{M}$ for such surfaces modulo birational equivalence was proven by Gieseker using GIT theory and the birationality of $5$-canonical map proven by Bombieri. It follows that for fixed values of $\chi(\mathcal{O}_S)$ and $K_S^2$ the space $\mathfrak{M}_{\chi, \, K^2}$ has a finite number of irreducible components.   

As in the case of $\mathcal{M}_g$, locally in a neighborhood of a point $[S]$ the moduli space $\mathfrak{M}$ is given by a quotient of the base $\textrm{Def}(S)$ of the Kuranishi family of $S$ by the finite group $\textrm{Aut}(S)$. However, in contrast with the case of curves, often patologies arise, for instence $\textrm{Def}(S)$ can be non reduced. This was first explained by Catanese in the case when $K_S$ is not ample, then Vakil later proved that such phenomena can be regarded as a particular case of a more general situation that he called the *Murphy's law* for moduli spaces.

In recent years, many people studied possible compactifications of $\mathfrak{M}_{\chi, \, K^2}$. For instance, Kollar and Shepherd Barron proposed to add surfaces with semi-log canonical singularities and ample $K_S$, that they called *stable surfaces* by analogy with stable curves. Using this construction, Alexeev and Pardini where for instance able to provide explicit compactifications of the moduli space of Burniat and Campedelli surfaces.

Here is a (short) bibliography on the subject:

D. Gieseker: *Global moduli for surfaces of general tipe*, Invent. math **43** (1977), 233-282.

F. Catanese: *On the moduli spaces of surfaces of general type*, J. Differential geometry **19** (1984), 483-515.

F. Catanese: *Everywhere non reduced moduli spaces*, Invent. math. **98** (1989), 293-310.

R: Vakil: *Murphy's Law in algebraic geometry: Badly-behaved deformation spaces*, Invent. Math. **164** (2006), 569-590. 

J. Kollar and N. I- Shepherd-Barron: *Threefolds and deformations of surface singularities*, Invent. Math. **91** (1988), 299-338.

V. Alexeev, R. Pardini: *Explicit compactifications of moduli spaces of Campedelli and Burniat surfaces*, arXiv:0901.4431.