It is not always the case that the composite of two operators is nice. In general, the composite of $S$ and $T$ is defined by $$ D(ST)=\{\xi\in D(T) | T\xi\in D(S)\}. $$ There is no reason to expect this space to be dense.
I agree with @Yemon. You want the affiliated operators. Here's an interesting parallel for finite measure spaces that I heard in a mini-course from Ozawa. Just as $$L^\infty(X,\mu)\subset L^1(X,\mu)\subset L^2(X,\mu)$$$$L^\infty(X,\mu)\subset L^2(X,\mu)$$ which is a subset of all measurable functions, if $(M,tr)$ is a finite von Neumann algebra, $$M=L^\infty(M,tr)\subset L^1(M,tr)\subset L^2(M,tr)\subset \eta(M),$$$$M=L^\infty(M,tr)\subset L^2(M,tr)\subset \eta(M),$$ the affiliated operators. The inclusion for $L^2(M)$ into $\eta(M)$ is given by $$ \xi\mapsto (L_\xi\colon m\mapsto \xi m), $$ (be careful - you need to take the closure of $L_\xi$. Clearly $L_\xi$ commutes with right multiplication by $U(M)$) and the image of $L^2(M)$ is the set of all $T\in \eta(M)$ such that if $$ |T|=\int t dE(t) $$ is the usual spectral measure, then $$ \int t^2 d tr(E(t)) <\infty. $$
About your comment in response to @Yemon: Indeed the trace works. If you have a positive function, use indicator functons to show the trace is semi-finite and faithful (see Takesaki I). Normal is clear. If you have a measurable function $f$ on $(X,\mu)$, construct a multiplication operator by letting $$ D(M_f)=\{\xi\in L^2(X,\mu)| f\xi \in L^2(X,\mu)\}. $$ If $\mu$ is finite, and $f$ is a measurable, real valued, a.e. finite function, then $M_f$ is self-adjoint, and the spectrum is the essential range of $f$.
Reed and Simon have a good introduction to unbounded operators, in particular the spectral theorem.
Somehow the brackets aren't showing up for the sets above, but I hope it's legible.