Relation between Gaussian processes and RKHSs with tensor product kernels

Question

For sets $\cal X$ and $\cal Y$, let $a:{\cal X}\times{\cal X}\rightarrow \mathbb{R}$ and $a:{\cal Y}\times{\cal Y}\rightarrow \mathbb{R}$ be positive definite symmetric kernels. Define the tensor product $a\otimes b$ as usual via $$ (a\otimes b)((x,x'),(y,y')) = a(x,x')a(y,y') $$ for all $x,x'\in \cal X$ and $y,y'\in \cal Y$.

A positive definite kernel defines both a Gaussian process (GP) and a reproducing kernel Hilbert space (RKHS), also known as the Cameron-Martin space of the GP (and defined as the set of shift vectors for the GP that preserve equivalence).

Define ${\cal R}_k$ as the RKHS with reproducing kernel $k$. Defining $$ {\cal R}_a\otimes{\cal R}_b = \langle\{f\otimes g|f\in{\cal R}_a,g\in{\cal R}_b,\}\rangle ,$$ the vector product tensor product, it is well-known that $$ {\cal R}_{a\otimes b} \subset {\cal R}_a\otimes{\cal R}_b $$ where the subset is strict in the infinite dimensional case, and \begin{equation}\tag{1}\label{closure} {\cal R}_{a\otimes b} = \overline{{\cal R}_a\otimes{\cal R}_b} \end{equation} where the closure is with respect to the tensor product norm.

My question is what can we say about the supports of the corresponding Gaussian processes? Denoting an appropriately defined support of a Gaussian process $GP(0,a)$ by ${\cal S}_a$, is it true that $$ {\cal S}_{a\otimes b} \subset {\cal S}_a\otimes{\cal S}_b $$ and if so is it possible to define some kind of closure operation so that an analogue of \eqref{closure} holds?

Background: GPs with tensor product kernels are used a lot in GP regression, so an answer could help to understand why it may make sense to use GPs with tensor product kernels.

Answer 1

For simplicity, I assume that the Gaussian process $g$ has mean zero such that it is completly specified by its covariance function $k$.

Note that the support of a (Borel) measure depends on the chosen topology. So I cannot give a general answer, which is independet of the chosen topology. For non-Borel measures, the question is probably much more complicated.

Let $\mathcal{F}$ be a topological $\mathbb{R}$-vectorspace of functions $X\to\mathbb{R}$, where $X\subseteq\mathbb{R}^d$. We assume furthermore that $\mathcal{F}$ is a Banach space (much of the following also holds for Frechet spaces, I think) and that all realizations of $g$ are contained in $\mathcal{F}$.

We have to distinguish Tensor products and closures.

Denote by

• $\operatorname{clo}_\mathcal{R}$ the closure in the category of Hilbert spaces.
• $\operatorname{clo}_\mathcal{F}$ the closure in the category of Banach spaces.

Denote by

• $\otimes_\mathbb{R}$ the tensor product in the category of $\mathbb{R}$-vectorspaces.
• $\otimes_\mathcal{R}$ the tensor product in the category of Hilbert spaces, which is the tensor product $\otimes_\mathbb{R}$ in the category of vector spaces followed by the closure in the Hilbert space, i.e. $\otimes_\mathcal{R}=\operatorname{clo}_\mathcal{R}\circ\otimes_\mathbb{R}$.
• $\otimes_\mathcal{F}$ the tensor product in the category of Banach spaces, which is the tensor product $\otimes_\mathbb{R}$ in the category of vector spaces followed by the closure in the Banach space, i.e. $\otimes_\mathcal{F}=\operatorname{clo}_\mathcal{F}\circ\otimes_\mathbb{R}$

Now, by Lemma 5.1 from ``Reproducing kernel Hilbert spaces of Gaussian priors'' (https://arxiv.org/abs/0805.3252v1) the closure (w.r.t. its own norm) $\operatorname{clo}_\mathcal{R}\mathcal{R}(k)$ of the RKHS $\mathcal{R}(k)$ of $k$ is the support $\operatorname{sup}(g)$ of the Gaussian process $g$ in $\mathcal{F}$.

As the question already mentions, given two covariance functions $k_1$ and $k_2$ with corresponding Gaussian processes $g_1$ and $g_2$, we have $\mathcal{R}(k_1\otimes k_2)=\mathcal{R}(k_1)\otimes_\mathcal{R}\mathcal{R}(k_2)$, where the first tensor product $k:=k_1\otimes k_2$ is just the pointwise product of the covariance functions $k_1$ and $k_2$. This implies for the Gaussian process $g$ with covariance $k$ that \begin{align*} \operatorname{supp}(g) &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1\otimes k_2)) \\ &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1)\otimes_R\mathcal{R}(k_2)) \\ &= \operatorname{clo}_\mathcal{F}(\operatorname{clo}_\mathcal{R}(\mathcal{R}(k_1)\otimes_\mathbb{R}\mathcal{R}(k_2))) \\ &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1)\otimes_\mathbb{R}\mathcal{R}(k_2)) \\ &= \mathcal{R}(k_1)\otimes_\mathcal{F}\mathcal{R}(k_2) \\ &= \operatorname{clo}_\mathcal{F}(\mathcal{R}(k_1))\otimes_\mathcal{F}\operatorname{clo}_\mathcal{F}(\mathcal{R}(k_2)) \\ &= \operatorname{supp}(g_1)\otimes_\mathcal{F}\operatorname{supp}(g_2) \end{align*}

This answers your question in the positive, at least for the above assumptions (mainly: $\mathcal{F}$ is Banach). Probably, your question still has a positive answer without out such assumptions.

Another good reference, which shows how complicated these questions are, is https://arxiv.org/abs/1604.05251.