Motivation
Let $(Z(x))_{x\in \mathbb{R}^n}$ be a random function (also known as random field, or random process), i.e. a collection of random variables, but also $Z\in C(\mathbb{R}^n)$ almost surely. The construction of this is not completely trivial, but let us take this as a given.
Now if one were to simulate such a random function at a discrete number of points $x_1,\dots, x_n$, one is interested in the marginal distribution, i.e. we want to know $$ \mathbb{E}[f(Z(x_1), \dots, Z(x_n))] $$ for all $f$. What now happens in Bayesian Optimization, is that we sample our random function at some point $x_0$, i.e. we obtain $Z(x_0)$, then we want to know the conditional distribution of $Z(x)$ given $Z(x_0)$. I.e. $$ \mathbb{E}[f(Z(x))\mid Z(x_0)] $$ This is not a huge problem if we assume $Z$ is a Guassian random function (Gaussian process) with known covariance function. But using this conditional distribution, we tend to select our next evaluation point $X_1$. I.e. $X_1$ is a random variable which is $\sigma(Z(x_0))$ measurable. If we now want to know the distribution of $$ \mathbb{E}[f(Z(x))\mid Z(X_1), Z(x_0)] $$ things become much more tricky, as $Z(X_1)$ is a much more complicated object than $Z(x_0)$. My hypothesis is that $$ \mathbb{E}[f(Z(x))\mid Z(X_1), Z(x_0)] = \bigl(y\mapsto \mathbb{E}[f(Z(x))\mid Z(y), Z(x_0)]\bigr)(X_1) $$ as $X_1$ is completely measurable with regards to $\sigma(Z(x_0)$.
Hypothesis
More generally I expect for a random function in $Z \in C(\mathbb{R}^n)$ $$ \mathbb{E}[f(Z(x))\mid Z(X_n),\dots, Z(X_1),Z(x_0)] = \bigl((y_n,\dots, y_1)\mapsto \mathbb{E}[f(Z(x))\mid Z(y_n),\dots, Z(y_1),Z(x_0)]\bigr)(X_n,\dots,X_1) $$ for $X_n$ measurable with regards to $$ \mathcal{F}_{n-1} = \sigma(Z(x_0),Z(X_1)\dots, Z(X_{n-1})). $$
Progress
I have been able to prove the claim:
Let $Z$ be an almost surely random function, $X$ be $\mathcal{F}$ measurable, then $$ \mathbb{E}[f(Z(X))\mid \mathcal{F}] = \Bigl(y\mapsto \mathbb{E}[f(Z(y))\mid \mathcal{F}]\Bigr)(X) $$
using the fact that the evaluation function $e(z,y) = z(y)$ is continuous and therefore measurable for continuous functions, and regular conditional distributions (I can post a proof if requested).
Now if we were not in the continuous case, we could do the following $$ \begin{aligned} &\mathbb{P}(Z(x) = z \mid Z(X_1) = z_1, Z(x_0) = z_0)\\ &\overset{\text{Bayes}}= \frac{\mathbb{P}(Z(X_1)=z_1 \mid Z(x)=z, Z(x_0)=z_0)\mathbb{P}(Z(x)=z, Z(x_0)=z_0)}{\mathbb{P}(Z(X_1)=z_1\mid Z(x)=z)\mathbb{P}(Z(x)=z)} \end{aligned} $$ and then apply the proven statement above since $X_1$ is $\sigma(Z(x), Z(x_0))$ measurable as it is already $\sigma(Z(x_0))$ measurable. Afterwards we simply use Bayes backwards to get $$ \begin{aligned} &\mathbb{P}(Z(x) = z \mid Z(X_1) = z_1, Z(x_0) = z_0)\\ &=\Bigl(y\mapsto \frac{\mathbb{P}(Z(y)=z_1 \mid Z(x)=z, Z(x_0)=z_0)\mathbb{P}(Z(x)=z, Z(x_0)=z_0)}{\mathbb{P}(Z(y)=z_1\mid Z(x)=z)\mathbb{P}(Z(x)=z)}\Bigr)(X_1)\\ &= \bigl(y \mapsto \mathbb{P}(Z(x) = z \mid Z(y) = z_1, Z(x_0) = z_0)\bigr)(X_1) \end{aligned} $$ Unfortunately, it is not quite so easy to translate this to the continuous case. $Z(x_0)\in A$ does not work, since we need to know its precise value to know $X_1$. Well, since we have that $Z$ is continuous, it might work like this. But I am unsure. Such approximating arguments seem to be very troublesome (cf. Borel-Kolmogorov Pradox)
Does anyone already know the solution to this problem, or has any idea how to approach this? Since this is related to a common practical usecase, it seems like this should be something someone has already looked at.
