Gaussian processes, sample paths and associated Hilbert space.

Question

Given a Gaussian process on some topological space $T$, with a continuous covariance kernel $C(\cdot,\cdot)\colon T\times T\to R$, we can associate a Hilbert space, which is the reproducing kernel Hilbert space of real-valued functions on $T$, with $C$ as kernel function. This contruction is given in, for instance, R J Adler & J E Taylor: "Random Fields and Geometry", and surely a lot of other places. We can suppose the topological space $T$ is separable.

A very rapid review of the construction is: Define an inner product space $H_0$ as consisting of all real-valued functions on $T$ of the form $f(x) = \sum_{i=1}^n a_i C(x_i,x)$, for real numbers $a_i$ and points in $T$, $x_i$. We can define an inner product on $H_0$ by $\left\langle\sum a_i C(x_i,\cdot), \sum b_j C(y_j,\cdot)\right\rangle = \sum \sum a_i b_j C(x_i,y_j)$.

Then the reproducing kernel Hilbert space associated with our gaussian process is the completion $H$ of $H_0$.

Now, this strongly suggests (to be usefull, and by the Karhunen-Loéve theorem, which is based on this construction) that sample paths of our Gaussian process belongs to H with probability 1. This must be proved somewhere, but where? Anybody knows a reference?

Answer 1

The question of continuity of a Gaussian process is a rich one with a lot of theory. Let $T$ be a compact index set, and suppose that $X_t$ is a mean-zero Gaussian process with covariance function $c(t,s)$. The continuity properties of the process $X_t$ are entirely determined by the covariance function.

One very simple condition uses the Kolmogorov continuity theorem. Let $d \ge 1$, and suppose that the index set $T$ is a compact subset of $\mathbb R^d$. Suppose that the covariance function $c$ satisfies $$c(t,t) - 2c(t,s) + c(s,s) \le C|t - s|^{d + \beta},$$ for some positive constants $C$ and $\beta$. Then the $d$-dimensional Kolmogorov theorem implies that, with probability one, there exists a continuous version of $X_t$ on $T$.

This is far from the most general sufficient condition for a Gaussian process to be continuous. Since you have a copy of Adler & Taylor handy, take a look at Section 1.3, Boundedness and Continuity.

Answer 2

No, it is not true for simple examples such as standard Brownian motion or a sequence of independent random variables.

1. Suppose $W$ is a standard Brownian motion on the interval $[0,T]$. The covariance kernel is $C(s,t)=\mathbb{E}[W_sW_t]=\min(s,t)$. Then, the associated reproducing kernel Hilbert space, $H$, is the set of absolutely continuous functions $f\colon[0,T]\to\mathbb{R}$ with $f(0)=0$ and $\int_0^T\left(\frac{df(t)}{dt}\right)^2\,dt < \infty$. However, sample paths of Brownian motion are nowhere differentiable with probability one, so have zero probability of being in $H$.
2. Suppose $X_1,X_2,\ldots$ is a sequence of independent standard normal random variables. Then, $T=\mathbb{N}$ and the covariance kernel is $C(m,n)=\mathbb{E}[X_mX_n]=1_{\{m=n\}}$. The reproducing kernel Hilbert space, $H$, is just $\ell^2(\mathbb{N})$. However, by the strong law of large numbers, $\sum_{m\le n}X_m^2$ grows at rate $n$, and $\sum_nX_n^2$ is almost surely infinite so, again, the process has paths with zero probability of being in $H$.

However, even though the Gaussian process ($X$, say) does not have sample paths lying in the Hilbert space $H$, what we can say is that it is described by a cylindrical measure on $H$. This means that it has consistently defined projections into the finite dimensional subspaces of $H$ or, equivalently, that we can consistently define the values of the "inner product" $\langle f, X\rangle$ for $f\in H$. In the first example above, where $X$ is a Brownian motion, we would have $$ \begin{align} \langle f,X\rangle &= \int_0^T\frac{\partial f(t)}{\partial t}\frac{\partial X(t)}{\partial t}\,dt\\ &=\int_0^T\frac{\partial f(t)}{\partial t}\,dX(t) \end{align} $$ The first expression on the right hand side does not make sense as $X$ is nowhere differentiable but, reinterpreting it as a stochastic integral with respect to $dX$, we get a meaningful expression. Similarly in the second example above, where $X_n$ is an IID sequence of standard normal variables, we can write $$ \langle f,X\rangle = \sum_{n=1}^\infty f(n)X(n). $$ Even though $X$ is not in $H=\ell^2(\mathbb{N})$, this sum converges (both in the $L^p$ norms for $1\le p < \infty$ and almost surely).

In fact, $X$ has the canonical Gaussian measure on $H$ and, as mentioned in the Wikipedia link, it is never a true measure when $H$ is infinite dimensional. I also answered another question concerning cylinder measures and the canonical Gaussian distribution (link) a while ago which could be of relevance to this question.

Answer 3

I came across the same question and found this thread. It doesn't look like a decisive conclusion was reached.

The reference you were looking for is this paper, particularly Theorem 7.4. If I understand it correctly, it says the sample path is in any slighter "larger" RKHS with probability 1. Hope this is still helpful.

Answer 4

Short answer:

If the RKHS $\mathcal{H}_C$ associated to the kernel of your GP is infinite dimensional, then sample paths are in $\mathcal{H}_C$ with probability 0.

Nevertheless, one can show that sample paths are contained in a "larger" RKHS, and Steinwart showed how to construct such an RKHS as a power of the original RKHS.

---

Sections 4.2 and 4.3 of Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences by Kanagawa et al. give a nive and brief account of sample path properties of gaussian processes and of how to construct an RKHS containing the paths of a GP.