Suppose $s∈(0,1)$, $D$ is an open set in $\mathbb{R}^d$. Define $$ H^s=(1−\Delta)^{-s/2}L^2\left(\mathbb{R}^d\right), $$ $$ H^s_D=\left\{f\in H^s:f=0 \mbox{ a.e. on } D^c\right\}. $$ Q: Is $C^\infty_c(D)$ dense in $H^s_D$(with norm $\|\cdot\|_{H^s}$) for any open set $D$?
Is there any element reference?
Assuming that the boundary of $D$ is regular, say $D$ is a bounded domain with Lipschitz boundary, the density is true. Equivalent definition of $H^2$ is as follows: $H^s=W^{s,2}$, where $$ \Vert f\Vert_{W^{2,s}}=\Vert f\Vert_2+\left(\int_{\mathbb{R}^d}\int_{\mathbb{R}^d} \frac{|f(x)-f(y)|^2}{|x-y|^{n+2s}}\right)^{1/2} $$ For a proof, see Section 3 in [1]. Now it is relatively easy to prove that a function in $W^{2,s}$ that vanishes in the complement of $D$ can be approximated by $C_0^\infty(D)$. This is more or less contained in Corollary 5.5 in [1].
I am pretty sure that for general domains there are counterexample. If you are interested in irregular domains I can try to find one.
[1] Di Nezza, E., Palatucci, G., Valdinoci, E., Hitchhiker's guide to the fractional Sobolev spaces.. Bull. Sci. Math. 136 (2012), no. 5, 521-573. (MathSciNet review).
