Question

Does any body know any reference in which the geometry of compactified moduli space of genus two curves ( Which is a three dimensional variety/stack/...) has been studied?

Answer 1

Genus 2 curves are hyperelliptic and so their coarse moduli space is just the Riemann-Hurwitz space $(\mathbb{P}^1)^6/(SL_2 \cdot S_6)$. So the description of $M_2$ is closedly linked with the invariants of binary sextic forms. The classic reference is the paper

J. Igusa, Arithmetic Variety of Moduli for Genus Two, Annals of Mathematics, Vol. 72, No. 3 (1960), pp. 612-649.

Brendan Hassett's paper Classical and minimal models of the moduli space of curves of genus two is also a nice paper studying explicit compactifications for $M_2$ and their birational geometry properties.

Answer 2

I recommend part III, the case g=2, of Mumford's ":Towards an enumerative geometry of the moduli space of curves", in the Shafarevich 60th birthday volume.

Answer 3

Over the complex numbers, you might be interested in: Mostafa:Die Singularitäten der Modulmannigfaltigkeit $\overline M_g(n)$ der stabilen Kurven vom Geschlecht $g\geq 2$ mit $n$-Teilungspunktstruktur. (German) [The singularities of the moduli variety $\overline M_{g}(n)$ of stable curves of genus $g\geq 2$ with $n$-division point structure] J. Reine Angew. Math. 343 (1983), 81–98.

Over a field of any characteristic, in my paper § 3, the scheme $\overline M_{2}$ over $\mathbb Z$ (and over any field $k$) is described as the normalization of the blowup of the weighted projective scheme $$\mathrm{Proj}\mathbb Z[J_2, J_4, J_6, J_8, J_{10}]/(J_4^2-J_2J_6+4J_8)$$ (the $J_i$'s are Igusa invariants and have weight $i$) along some explicit center. The singularities over $k$ are described as well.