# Relation between Gaussian processes and RKHSs with tensor product kernels

For sets $$\cal X$$ and $$\cal Y$$, let $$a:{\cal X}\times{\cal X}\rightarrow \mathbb{R}$$ and $$a:{\cal Y}\times{\cal Y}\rightarrow \mathbb{R}$$ be positive definite symmetric kernels. Define the tensor product $$a\otimes b$$ as usual via $$(a\otimes b)((x,x'),(y,y')) = a(x,x')a(y,y')$$ for all $$x,x'\in \cal X$$ and $$y,y'\in \cal Y$$.

A positive definite kernel defines both a Gaussian process (GP) and a reproducing kernel Hilbert space (RKHS), also known as the Cameron-Martin space of the GP (and defined as the set of shift vectors for the GP that preserve equivalence).

Define $${\cal R}_k$$ as the RKHS with reproducing kernel $$k$$. Defining $${\cal R}_a\otimes{\cal R}_b = \langle\{f\otimes g|f\in{\cal R}_a,g\in{\cal R}_b,\}\rangle ,$$ the vector product tensor product, it is well-known that $${\cal R}_{a\otimes b} \subset {\cal R}_a\otimes{\cal R}_b$$ where the subset is strict in the infinite dimensional case, and $$$$\tag{1}\label{closure} {\cal R}_{a\otimes b} = \overline{{\cal R}_a\otimes{\cal R}_b}$$$$ where the closure is with respect to the tensor product norm.

My question is what can we say about the supports of the corresponding Gaussian processes? Denoting an appropriately defined support of a Gaussian process $$GP(0,a)$$ by $${\cal S}_a$$, is it true that $${\cal S}_{a\otimes b} \subset {\cal S}_a\otimes{\cal S}_b$$ and if so is it possible to define some kind of closure operation so that an analogue of \eqref{closure} holds?

Background: GPs with tensor product kernels are used a lot in GP regression, so an answer could help to understand why it may make sense to use GPs with tensor product kernels.

• Is the tensor product with ${\cal R}_{a\otimes b} \subset {\cal R}_a\otimes{\cal R}_b$ the tensor product in the category of vector spaces? Because, if you take the tensor product in the category of Hilbert spaces (which is the composition of the vector space tensor product, followed by the closure monad), then ${\cal R}_{a\otimes b} = {\cal R}_a\otimes{\cal R}_b$. – Markus Lange-Hegermann Jan 8 '20 at 12:23
• If I remember correctly, the support of a GP was just the closure of its RKHS. This closure depends on the topology. Then, one would "just" need to check which tensor product is the correct one in the category where the supports live (Banach spaces, Frechet spaces, ...) and check whether the closure RKHS$\to$support commutes with the tensor product. This might again depend on the topology. – Markus Lange-Hegermann Jan 8 '20 at 12:31
• Thanks for the comments, I clarified the tensor product definition in the post (so this should be in the category of vector spaces). – Wicher Jan 8 '20 at 12:40
• Thanks, that seems to be the right direction (support of a GP is the closure of its RKHS), but I do not intuit this at all yet. According to Lukic and Beder (ams.org/journals/tran/2001-353-10/S0002-9947-01-02852-5/…), in the infinite dimensional case the probability that a sample path of a GP lies in the corresponding RKHS is zero. So some definitions of support of the GP (say, roughly speaking, the smallest set such that sample paths are in it with probability one) should exclude the RKHS I guess... – Wicher Jan 8 '20 at 12:49
• The RKHS is dense in the support but still has measure zero. This is similar to the rationals, which are dense in the support of the standard Gaussian but still have measure zero. – Markus Lange-Hegermann Jan 10 '20 at 15:39

For simplicity, I assume that the Gaussian process $$g$$ has mean zero such that it is completly specified by its covariance function $$k$$.

Note that the support of a (Borel) measure depends on the chosen topology. So I cannot give a general answer, which is independet of the chosen topology. For non-Borel measures, the question is probably much more complicated.

Let $$\mathcal{F}$$ be a topological $$\mathbb{R}$$-vectorspace of functions $$X\to\mathbb{R}$$, where $$X\subseteq\mathbb{R}^d$$. We assume furthermore that $$\mathcal{F}$$ is a Banach space (much of the following also holds for Frechet spaces, I think) and that all realizations of $$g$$ are contained in $$\mathcal{F}$$.

We have to distinguish Tensor products and closures.

Denote by

• $$\operatorname{clo}_\mathcal{R}$$ the closure in the category of Hilbert spaces.
• $$\operatorname{clo}_\mathcal{F}$$ the closure in the category of Banach spaces.

Denote by

• $$\otimes_\mathbb{R}$$ the tensor product in the category of $$\mathbb{R}$$-vectorspaces.
• $$\otimes_\mathcal{R}$$ the tensor product in the category of Hilbert spaces, which is the tensor product $$\otimes_\mathbb{R}$$ in the category of vector spaces followed by the closure in the Hilbert space, i.e. $$\otimes_\mathcal{R}=\operatorname{clo}_\mathcal{R}\circ\otimes_\mathbb{R}$$.
• $$\otimes_\mathcal{F}$$ the tensor product in the category of Banach spaces, which is the tensor product $$\otimes_\mathbb{R}$$ in the category of vector spaces followed by the closure in the Banach space, i.e. $$\otimes_\mathcal{F}=\operatorname{clo}_\mathcal{F}\circ\otimes_\mathbb{R}$$

Now, by Lemma 5.1 from Reproducing kernel Hilbert spaces of Gaussian priors'' (https://arxiv.org/abs/0805.3252v1) the closure (w.r.t. its own norm) $$\operatorname{clo}_\mathcal{R}\mathcal{R}(k)$$ of the RKHS $$\mathcal{R}(k)$$ of $$k$$ is the support $$\operatorname{sup}(g)$$ of the Gaussian process $$g$$ in $$\mathcal{F}$$.

As the question already mentions, given two covariance functions $$k_1$$ and $$k_2$$ with corresponding Gaussian processes $$g_1$$ and $$g_2$$, we have $$\mathcal{R}(k_1\otimes k_2)=\mathcal{R}(k_1)\otimes_\mathcal{R}\mathcal{R}(k_2)$$, where the first tensor product $$k:=k_1\otimes k_2$$ is just the pointwise product of the covariance functions $$k_1$$ and $$k_2$$. This implies for the Gaussian process $$g$$ with covariance $$k$$ that \begin{align*} \operatorname{supp}(g) &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1\otimes k_2)) \\ &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1)\otimes_R\mathcal{R}(k_2)) \\ &= \operatorname{clo}_\mathcal{F}(\operatorname{clo}_\mathcal{R}(\mathcal{R}(k_1)\otimes_\mathbb{R}\mathcal{R}(k_2))) \\ &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1)\otimes_\mathbb{R}\mathcal{R}(k_2)) \\ &= \mathcal{R}(k_1)\otimes_\mathcal{F}\mathcal{R}(k_2) \\ &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1))\otimes_\mathcal{F}\operatorname{clo}_\mathcal{F}(\mathcal{R}(k_2)) \\ &= \operatorname{supp}(g_1)\otimes_\mathcal{F}\operatorname{supp}(g_2) \end{align*}

This answers your question in the positive, at least for the above assumptions (mainly: $$\mathcal{F}$$ is Banach). Probably, your question still has a positive answer without out such assumptions.

Another good reference, which shows how complicated these questions are, is https://arxiv.org/abs/1604.05251.

• Many thanks! Just two queries if I may. For the fourth equality, so if $\cal F$ is a Banach space and $\cal R$ an RKHS, then $\text{clo}_{\cal F}\circ \text{clo}_{\cal R}=\text{clo}_{\cal F}$? Similarly the sixth equality, this is valid because the closure of the tensor product of two spaces is the closure of the tensor product of their closures? – Wicher Jan 12 '20 at 18:19
• @Wicher: yes, you are right. – Markus Lange-Hegermann Jan 12 '20 at 18:38