The following might be quite straightforward, but I very rarely work in detail with unbounded operators, so I thought it would be worth seeing quickly if I have overlooked an example that is obvious from the right point of view.
Let $H=L^2(-\infty,\infty)$ and let $S:H \supset {\rm dom}(S) \to H$ be the densely-defined operator $(Sf)(x)=e^xf(x)$, with domain $$ {\rm dom}(S)=\{ f\in L^2(-\infty,\infty) \colon \int_{-\infty}^\infty e^{2x} |f(x)|^2\,dx <\infty \} $$ Then $S$ is self-adjoint.
Question. Does there exists a projection $P:H\to H$, with range $V$, satisfying the following properties?
$V$ is infinite-dimensional and $V':={\rm dom}(S)\cap V$ is dense in $V$.
there is a compact operator $R:V\to V$ such that $RPSP\vert_{V'}= I_{V'}$?
Some remarks:
(a) If we weakened 2. to merely require $R$ being bounded, then this should be easy by taking $V=L^2[0,\infty)$ and $R:V\to V$ to be multiplication by $e^{-x}$. But of course $R$ has continuous spectrum, so it can't be compact.
(b) If we intertwine with the Fourier transform or something similar, perhaps we can we obtain such a $V$ as the solution space to a suitable differential equation? (I have not looked at this vague idea in any detail yet.)
(c) If the answer to the question is positive, can we even get $R$ being trace-class?