**The revised question**

After more reflection on the problem, I might have found the answer by myself. Let $U$ be an open subset of $M$, irrespective of whether it has a boundary or not. Let

$$\mathcal D _\infty ' = \{ u \in \mathcal D ' (U) \mid \forall \varepsilon >0 \ \exists K _\varepsilon \ \text{compact such that} |\langle u, \varphi \rangle| < \varepsilon \\ \forall \varphi \in \mathcal D (U) \ \text{with supp} \ \varphi \cap K _\varepsilon = \emptyset \ \text{and} \ \sup \limits _\alpha \sup \limits _U |\partial _\alpha \varphi| \le 1 \} .$$

Note that if $u \in \mathcal D _\infty ' (M)$, then $\partial _\alpha u \in \mathcal D _\infty ' (M) \forall \alpha$ and that $\mathcal D _\infty ' (M)$ contains all the smooth functions $f$ such that $\partial _\alpha f$ vanishes at infinity $\forall \alpha$. It also contains all the compactly-supported distributions, which suggests that the given definition is a good one.

A new interesting question arises, though: does $\mathcal D _\infty ' (M)$ have a nice predual? In order for the question to make sense, I should specify the topology on $\mathcal D ' (M)$ (the weak* or the strong one). I do not know which one to choose, I guess the question could be reformulated as: does any of these two topologies (or any other one) give a nice predual?

**The original question**

If $U$ is an open subset in a Riemannian manifold $M$ (as a first step, $M = \Bbb R^n$ should suffice) with the boundary $\partial U$ a submanifold in $M$, does there exist something as *"the space of distributions from $\mathcal D ' (U)$ that vanish on $\partial U$"*?

The closest thing that comes to mind is the space of those distributions with the support (not necessarily compact) included in $U$, but this is needlessly restrictive for the problem (see rationale below).

The other thing that comes to mind is that this space should be the distributional-theoretic counterpart of the Sobolev spaces $W ^{k,p} _0 (U)$. If a "trace operator" $T : \mathcal D ' (U) \to \mathcal D ' (\partial U)$ existed, then this space could be defined as the kernel of $T$.

**Rationale**

Suppose that we want to show the uniqueness of the solution of the heat equation in the space of smooth functions that vanish on the boundary. The argument is a classic: under the assumption that the solution $u$ is real and $u(0, \cdot) = 0$, if $I(t) = \int u^2$ then $I'(t) = \int 2 \partial _t u \ u = \int 2 \Delta u \ u = -2 \int \| \nabla u \| ^2 \le 0$, so $I = I(0) = 0$, so $u = 0$. One also sees that $u$ cannot be complex since both its real and ist imaginary part would satisfy the equation and thus be $0$.

The core thing used above is the integration by parts in which $\int \Delta (u^2) = 0$ because $u$ vanishes on the boundary. I wonder whether this argument can be mimicked if instead of smooth functions one used "distributions vanishing on the boundary", provided that such a thing exist. Of course, I wouldn't know what to replace all the products of distributions that would show up with, but this is a different matter.

distributionswith compact support? For Sobolev spaces, one starts withfunctionsof compact support, not with generalized functions of compact (essential) support. Related to my question (should I ask it separately?), could one construct a theory of distributions, parallel to the usual one, but using the bounded smooth functions that vanish on the boundary (and drop the compact subsets from the seminorms) as test functions? $\endgroup$ – Alex M. Sep 22 '15 at 19:45