Kernel of the composition of operators

Question

Let $X \subset \mathbb{R}^{n}$, $Y \subset \mathbb{R}^{m}$, and $Z \subset \mathbb{R}^{p}$ be open subsets, and let $K_P \in C_0^\infty(X \times Y)$ and $K_Q \in C_0^{\infty}(Y \times Z)$. Then, $K_P$ and $K_Q$ define sequentially continuous linear mappings $P:C_0^\infty(Y) \longrightarrow C_0^\infty(X)$ and $Q:C_0^{\infty}(Z) \longrightarrow C_0^{\infty}(Y)$. Then, $P \circ Q: C_0^\infty(Z) \longrightarrow C_0^\infty(X)$ is a sequentially continuous linear mapping with kernel \begin{equation} K_{P \circ Q}=\int_{Y}K_P(\cdot,y)K_{Q}(y,\cdot)dy. \tag{*} \end{equation} Indeed, \begin{eqnarray*} \langle P \circ Q u,v \rangle_{\mathcal{D}'(X), \mathcal{D}(X)} &=& \langle K_P, v \otimes Qu \rangle_{\mathcal{D}'(X \times Y), \mathcal{D}(X) \otimes \mathcal{D}(Y)}\\&=& \int_{X \times Y} K_{P}(x,y)v(x)Qu(y)dx dy\\ &=& \int_{Y} Qu(y) \int_{X} K_P(x,y) v(x) dx dy\\ &=& \langle Qu, \int_{X} K_P(x,\cdot)v(x)dx \rangle_{\mathcal{D}'(Y), \mathcal{D}(Y)}\\ &=& \langle K_Q, \int_{X}K_P(x,\cdot)v(x)dx \otimes u \rangle_{\mathcal{D}'(Y \times Z), \mathcal{D}(Y) \otimes \mathcal{D}(Z)}\\ &=& \int_{Y \times Z} K_Q(y,z) \int_{X} K_P(x,y)v(x)dx u(z)dydz\\ &=& \int_{X \times Z} \int_{Y} K_{Q}(y,z)K_P(x,y)dyv(x)u(z)dxdz\\ &=& \langle \int_{Y}K_P(\cdot,y)K_Q(y,\cdot)dy, v \otimes u\rangle_{\mathcal{D}'(X \times Z), \mathcal{D}(X) \otimes \mathcal{D}(Z)}. \end{eqnarray*}

I would like to know what would be a generalization of $(*)$ for the kernel of the composition of operators with kernels in $\mathcal{D}'(X \times Y)$ and $\mathcal{D}'(Y \times Z)$. A priori it makes no sense to compose the operators generated by $K_P$ and $K_Q$ because they define operators (by the Schwartz Kernel Theorem) $P: \mathcal{D}(Y) \longrightarrow \mathcal{D}'(X)$ and $Q: \mathcal{D}(Z) \longrightarrow \mathcal{D}'(Y)$. So it may be necessary to impose some more conditions on the kernels (Maybe we should assume that $\hbox{supp } K_P$ and $\hbox{supp } K_Q$ are proper, in order to have properly supported operators). I also don't know if we can define the kernel as the integral, because in integrating we would have the product of two distributions.

Is there any book, article, that addresses this subject from a more general point of view (even if it is in the context of pseudodifferential operators)?

Answer 1

Following the Abdelmalek Abdesselam's suggestions I found the following result in J. J. Duistermaat's book Fourier Integral Operators (page 22). However, the proof is very succinct and I could not understand. I could not even understand why I can define $\Delta^*$ since the results for the existence of $\Delta^*$ require that the function be a difeomorphism or a submersion.

In the following theorem concerning continuous mappings $A: C_0^\infty(Y) \longrightarrow \mathcal{D}'(X)$ with distribution kernel $K_A \in \mathcal{D}'(X \times Y)$ it is conveniente to introduce the following notations. \begin{equation*} WF'(A)=\{((x,\xi),(y,\eta)) \in (T^*(X)\times T^*(Y)) \setminus 0: (x,y;\xi,-\eta) \in WF(K_A) \}, \end{equation*} \begin{equation*} WF'_X(A)=\{ (x,\xi) \in T^*(X)\setminus 0: \exists y \in Y: (x,y;\xi,0) \in WF(K_A) \} \end{equation*} \begin{equation*} WF'_Y(A)=\{ (y,\eta) \in T^*(Y)\setminus 0: \exists x \in X: (x,y;0,\eta) \in WF(K_A) \} \end{equation*} If $R_1 \subset U \times V$, $R_2 \subset V \times W$ are relations then the composition $R_1 \circ R_2 \subset U \times W$ is defined by \begin{equation*} R_1 \circ R_2= \{(u,w) \in U \times W: \exists v \in V: (u,v) \in R_1 \hbox{ and } (v,w) \in R_2 \}. \end{equation*}

Theorem 1.3.7. Let $X,Y,Z$ be $C^\infty$ manifolds, $A$ and $B$ continuous linear mappings: $C_0^\infty(Y) \longrightarrow \mathcal{D}'(X)$ and $C_0^\infty(Z) \longrightarrow \mathcal{D}'(Y)$, respectively. If $WF'_Y(A) \cap WF'_Y(B)= \emptyset$ and the projection into $X \times Z$ from the diagonal is $X \times Y \times Y \times Z$ a proper mapping, then $A \circ B$ is a well-defined continuous linear mapping: $C_0^\infty(Z) \longrightarrow \mathcal{D}'(X)$. Moreover $\hbox{supp} K_{A \circ B} \subset \hbox{supp} K_{A } \circ \hbox{supp} K_{B}$ and \begin{equation*} WF'(A \circ B) \subset WF'(A) \circ WF'(B) \cup (WF'_X(A) \times O_{T^*(Z)}) \cup ( O_{T^*(X)} \times WF'_Z(B)). \end{equation*} Proof: $K_{A \circ B}= \pi_* \Delta^* (K_A \otimes K_B)$, where $\Delta: (x,y,z) \mapsto (x,y,y,z): X \times Y \times X \longrightarrow X \times Y \times Y \times Y \times Z$ and $ \pi: (x,y,z) \mapsto (x,z): X \times Y \times Z \longrightarrow X \times Z$. $\blacksquare$

I think we would have to do a calculation like:

\begin{eqnarray} \langle \pi_* \Delta^* (K_A \otimes K_B), v \otimes u \rangle_{\mathcal{D}'(X \times Z), \mathcal{D}(X)\otimes \mathcal{D}(Z) } &=& \langle \Delta^* (K_A \otimes K_B), \pi^*(v \otimes u) \rangle_{\mathcal{D}'(X \times Y \times Z), \mathcal{D}(X)\otimes \mathcal{D}(Y)\otimes\mathcal{D}(Z) }\\ &\vdots&\\ &=& \langle K_A, v \otimes Bu \rangle_{\mathcal{D}'(X \times Y), \mathcal{D}(X)\otimes \mathcal{D}(Y) }\\ &=& \langle A(B(u)), v \rangle_{\mathcal{D}'(X ), \mathcal{D}(X)}, \end{eqnarray} but we would still have to ensure that $Bu \in \mathcal{D}(Y)$ and prove the other statements of the theorem..