Core for a Sobolev space Let $D$ be a domain of $\mathbb{R}^d$. That is, $D$ is a connected open subset of $\mathbb{R}^d$. The first-order Sobolev space $W^{1,2}(D)$ on $D$ is defined by
\begin{align*}
W^{1,2}(D)=\{f \in L^2(D,m) \mid \partial f/\partial x_i \in L^2(D,m),\, 1\le i \le d\}.
\end{align*}
Here, $m$ is the Lebesgue measure and $ \partial f/\partial x_i$ is the distributional derivative of $f$.
It is well known that $W^{1,2}(D)$ becomes a Hilbert space. The norm is defined by $$\|f\|_{W^{1,2}(D)}:=\left[\int_{D}\{f(x)^2+\sum_{i=1}^d (\partial f/\partial x_i)^2\}\,m(dx) \right]^{1/2}.$$

When are smooth functions $C^\infty_{c}({\overline{D}})(=C^\infty_{c}(\mathbb{R}^d)|_{\overline{D}})$ dense in $W^{1,2}(D)$ ?

If there is a bounded linear operator $T\colon W^{1,2}(D) \to W^{1,2}(\mathbb{R}^d)$ such that $Tf=f$, $m$-a.e. on $D$, we can easily check that $C^\infty_{c}({\overline{D}})$ becomes a dense subspace of $W^{1,2}(D)$ (because $C^\infty_{c}(\mathbb{R}^d)$ is a dense subspace of $W^{1,2}(\mathbb{R}^d)$). Such an operator is called an extension operator, and it seems that its existence is known even when the boundary of $D$ is very complicated [for example, the Koch snowflake domain].

Can $C^\infty_{c}({\overline{D}})$ become dense in $W^{1,2}(D)$ without the extension operator?

I don't know such an example (of domains), so if anyone knows, please let me know.
 A: Smooth functions $C^\infty(D)$ are always dense in $W^{1,p}(D)$ when $1\leq p<\infty$. This is a classical result of Meyers and Serrin and you can find it in any textbook on Sobolev spaces. However, the functions are smooth in the interior and their befaviour at the boundary can be pretty bad, if the domain is bad.

Can $C^\infty_{c}({\overline{D}})$ become dense in $W^{1,2}(D)$ without the extension operator?

Yes it can! The following result is Corollary 1.2 in 1:

Theorem. If $\Omega$ is a planar Jordan domain, then $C^\infty(\mathbb{R}^2)$ is dense in $W^{1,p}(\Omega)$ for any $1\leq p <\infty$.

The result is pretty surprising, since the boundary of a planar Jordan domain can have positive 2-dimensional Lebesgue measure. This and many other Jordan domains are not a Sobolev extension domains.

When smooth functions $C^\infty_{c}({\overline{D}})(=C^\infty_{c}(\mathbb{R}^d)|_{\overline{D}})$ are dense in $W^{1,2}(D)$ ?

There are no known characterizations of such a domians and I believe that perhaps except for planar dominas it will not be possible to find such a characterization.
1 P. Koskela, Y. R.-Y. Zhang,
A density problem for Sobolev spaces on planar domains.
Arch. Ration. Mech. Anal. 222 (2016), no. 1, 1–14.
arXiv
A: (Too long for a comment.)
I have just learned that my colleagues, Bartłomiej Dyda and Michał Kijaczko, wrote a paper [1] on that particular problem for fractional Sobolev spaces. In their work, they cite Theorem 3.25 in McLean's book [2], which reads as follows (with the original notation):
Theorem: For any open set $\Omega$ and any real $s \geqslant 0$, the set $W^s(\Omega) \cap \mathcal E(\Omega)$ is dense in $W^s(\Omega)$.
So at least smooth functions in $\Omega$ are always dense in $W^s(\Omega)$ (but of course they need not extend smoothly to the boundary).
I did not have time to check carefully the two references mentioned above for an answer to your question — perhaps it is written somewhere in [2].
References:

*

*[1] B. Dyda, M. Kijaczko, On density of smooth functions in weighted fractional Sobolev spaces, Nonlinear Anal. 205 (2021): 112231, DOI:10.1016/j.na.2020.112231, arXiv:2009.00077

*[2] W. McLean, Strongly elliptic systems and boundary integral equations. Cambridge University Press, Cambridge, 2000.

