Consider $PGL_n$ acting by conjugation on the space of $n\times n$ matrices $M_n$, and let the GIT quotient map be $\pi:M_n\to M_n//PGL_n$. I think you are asking about the geometry of the fibre of zero.
Regardless, I believe this is a special case of the nullcone for a reductive group action on a vector space $V$.
Two papers that come to mind about the geometry of the nullcone are:
- A Stratification of the Null Cone Via the Moment Map by Linda Ness (appendix by David Mumford), American Journal of Mathematics, Vol. 106, No. 6 (Dec., 1984), pp. 1281-1329
- Irreducible components of the nullcone by Richardson, R. W., Invariant theory, 409–434, Contemp. Math., 88, Amer. Math. Soc., Providence, RI, 1989.
The work of these two authors is worth reading if you are interested in these kinds of problems (and the references therein).