A space of distributions vanishing on the boundary The revised question
After more reflection on the problem, I might have found the answer by myself. Let $U$ be an open subset of $M$, irrespective of whether it has a boundary or not. Let
$$\mathcal D _\infty ' = \{ u \in \mathcal D ' (U) \mid \forall \varepsilon >0 \ \exists K _\varepsilon \ \text{compact such that} |\langle u, \varphi \rangle| < \varepsilon \\ \forall \varphi \in \mathcal D (U) \ \text{with supp} \ \varphi \cap K _\varepsilon = \emptyset \ \text{and} \ \sup \limits _\alpha \sup \limits _U |\partial _\alpha \varphi| \le 1 \} .$$
Note that if $u \in \mathcal D _\infty ' (M)$, then $\partial _\alpha u \in \mathcal D _\infty ' (M) \forall \alpha$ and that $\mathcal D _\infty ' (M)$ contains all the smooth functions $f$ such that $\partial _\alpha f$ vanishes at infinity $\forall \alpha$. It also contains all the compactly-supported distributions, which suggests that the given definition is a good one.

A new interesting question arises, though: does $\mathcal D _\infty ' (M)$ have a nice predual? In order for the question to make sense, I should specify the topology on $\mathcal D ' (M)$ (the weak* or the strong one). I do not know which one to choose, I guess the question could be reformulated as: does any of these two topologies (or any other one) give a nice predual?

The original question
If $U$ is an open subset in a Riemannian manifold $M$ (as a first step, $M = \Bbb R^n$ should suffice) with the boundary $\partial U$ a submanifold in $M$, does there exist something as "the space of distributions from $\mathcal D ' (U)$ that vanish on $\partial U$"?
The closest thing that comes to mind is the space of those distributions with the support (not necessarily compact) included in $U$, but this is needlessly restrictive for the problem (see rationale below).
The other thing that comes to mind is that this space should be the distributional-theoretic counterpart of the Sobolev spaces $W ^{k,p} _0 (U)$. If a "trace operator" $T : \mathcal D ' (U) \to \mathcal D ' (\partial U)$ existed, then this space could be defined as the kernel of $T$.
Rationale
Suppose that we want to show the uniqueness of the solution of the heat equation in the space of smooth functions that vanish on the boundary. The argument is a classic: under the assumption that the solution $u$ is real and $u(0, \cdot) = 0$, if $I(t) = \int u^2$ then $I'(t) = \int 2 \partial _t u \ u = \int 2 \Delta u \ u = -2 \int \| \nabla u \| ^2 \le 0$, so $I = I(0) = 0$, so $u = 0$. One also sees that $u$ cannot be complex since both its real and ist imaginary part would satisfy the equation and thus be $0$.
The core thing used above  is the integration by parts in which $\int \Delta (u^2) = 0$ because $u$ vanishes on the boundary. I wonder whether this argument can be mimicked if instead of smooth functions one used "distributions vanishing on the boundary", provided that such a thing exist. Of course, I wouldn't know what to replace all the products of distributions that would show up with, but this is a different matter.
 A: Here's a suggestion. For simplicity, assume that $\partial U$ is the zero level set of a scalar function $f$, for which $0$ is a regular value. Let $C^\infty_b(U)$ be the space of smooth functions in $U$ whose derivatives of all orders extend continuously to $\overline{U}$. Make it into a Fréchet space by using sup-seminorms over compacts in $\overline{U}$. Let $C^\infty_f(U) \subset C^\infty(U)$ be the pre-image of $C^\infty_b(U)$ with respect to the multiplication-by-$f$ map, $C^\infty(U) \stackrel{f\cdot}{\longrightarrow} C^\infty(U) \supset C^\infty_b(U)$. Use the initial topology with respect to the map $C^\infty_f(U) \stackrel{f\cdot}{\longrightarrow} C^\infty_b(U)$. Let $\mathcal{D}_f(U) = \bigcup_{k=1,2,\ldots} C^\infty_{f^k}(U)$ and give it the inductive limit topology.
You can interpret $\mathcal{D}_f(U)$ as the set of smooth functions on $U$ that grow at most polynomially near $\partial U$. Then, I think $\mathcal{D}'_f(U)$ will be your desired space of distributions vanishing (to arbitrary order) on $\partial U$.
A: If $P$ is a linear partial differential operator in $\overline{U}$ and if the boundary $\partial U$ is non-characteristic for $P$, then every extendible distribution solution $u$ of $Pu=0$ in $U$ belongs, in a boundary collar $0\leq x<\varepsilon$, to $C_x^\infty([0,\varepsilon[,\mathcal{D}'(\partial U))$. This implies that boundary traces are defined by evaluating at $x=0$. (Here $x$ is a defining function; the same as $f$ in Igor Khavkine's answer.) This result is due to Peetre, and it can be found under the label partial hypoellipticity at the boundary in Hörmander's 1963 book. Melrose's microlocal theory of boundary problems (Acta Math. (1981)) gives a class of distributions on a manifold with boundary such that a canonical restriction (or boundary trace) map is defined; see Proposition 18.3.21 in Hörmander's volume III. Peetre's theorem is given an invariant generalization by Melrose.
So, regarding your original question, there is a natural space of distributions vanishing (to first order) at the boundary, the null space of Melrose's restriction map. As for your rationale: the space-time boundary is non-characteristic for the heat operator. So boundary values (and their vanishing or not) are well-defined. As for the boundary terms in integration by parts it may be useful to look at the theory of the Calderon projector for boundary problems.
A: $\def\bbR{\mathbb R}\def\bbN{\mathbb N}\def\sp{\kern.4mm}$Working locally in a manifold, one can transport the problem to $\bbR\times\bbR^N$ for some $N\in\bbN$ if $n\ge 2\,$. Then a distribution on $\Omega=\bbR^+\kern-.9mm\times\bbR^N$ having boundary values zero on $\{\sp 0\sp\}\times\bbR^N$ can be defined to be any $T$ in $\mathscr D'(\Omega)$ with the following property: There is a function $c:\bbR^+\to\mathscr D_\sigma'(\bbR^n)$ where the latter space is equipped with the weak$^*$ topology, with limit zero at $0\sp$, and such that $\bbR^+\owns t\mapsto c(t)\,(\varphi(\sp t\sp,\cdot\sp))$ being $L ^1(\bbR^+)$ with $T(\varphi)=\int_{\,\bbR^+}c(t)\,(\varphi(\sp t\sp,\cdot\sp))\,\mathrm d\sp t$ hold for all $\varphi$ in $\mathscr D(\Omega)\sp$. This, however, is only one possibility, and whether it is usefull or not depends on the context in which one is working.
