What's the probability distribution of a deterministic signal or how to marginalize dynamical systems? (functional integrals in probability theory)

Question

Because I still have no idea how it is possible for me to write down seemingly important equations ... that don't make any sense (at least for me) and because I haven't got any helpful comment so far, I'll be happy to offer a +100 bounty (that is, almost all my reputation!) not for a definitive anwser to this weird question but only for any serious, relevant feedback, thought, opinion or advice.

In many signal processing calculations, the prior probability distribution of the theoretical signal of interest (not the noisy experimental signal) is required. Here is the concrete problem from which my question has arisen:

Is there a Bayesian theory of deterministic signal? Prequel and motivation for my previous question

In random signal theory (à la Shannon), this probability distribution typically is a stochastic process, e.g. an i.i.d. stochastic process if you are a frequentist or an exchangeable stochastic process if you are a Bayesian, in the most basic cases.

What do such probability distributions become in deterministic signal theory/dynamical system theory?, that is the question.

To make it simple, consider a discrete-time real deterministic signal

$ s\left( {1} \right),s\left( {2} \right),...,s\left( {M} \right) $

For instance, it can be obtained by sampling a continuous-time real deterministic signal.

By the standard definition of a discrete-time deterministic dynamical system, there exists:

• a phase space $\Gamma$, e.g. $\Gamma \subset \mathbb{R} {^d}$, $\Gamma = \left[ {0,1} \right]$, etc.

• a state-space equation $f:\Gamma \to \Gamma $ such as $z\left( {m + 1} \right) = f\left[ {z\left( m \right)} \right]$;

• an output or observation equation $g:\Gamma \to \mathbb{R}$ such as $s\left( m \right) = g\left[ {z\left( m \right)} \right]$;

• an initial condition $ z\left( 1 \right)\in \Gamma $ in the domain of definition of $f$.

Hence, by definition we have

$\left[ {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right] = \left\{ {g\left[ {z\left( 1 \right)} \right],g\left[ {f\left( {z\left( 1 \right)} \right)} \right],...,g\left[ {{f^{M - 1}}\left( {z\left( 1 \right)} \right)} \right]} \right\}$

or, in probabilistic notations

$p\left[ {\left. {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right|z\left( 1 \right),f,g,\Gamma ,d} \right] = \prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}} $

Therefore, by "total probability and the product rule", the "marginal joint prior probability distribution" for a discrete-time deterministic signal conditional on phase space $\Gamma$ and its dimension $d$ formally/symbolically writes

$p\left[ {\left. {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right|\Gamma ,d} \right] = \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}p\left( {z\left( 1 \right),f,g} \right)} } } } $

Should phase space $\Gamma$ and its dimension $d$ be also unknown a priori, they should be marginalized as well so that the most general "marginal prior probability distribution" for a discrete-time deterministic signal I'm considering formally/symbolically writes

$p\left[ {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right] = \sum\limits_{d = 1}^{ + \infty } {\int\limits_{\wp \left( {{\mathbb{R}^d}} \right)} {{\text{D}}\Gamma \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}p\left( {z\left( 1 \right),f,g,\Gamma ,d} \right)} } } } } } $

where ${\wp \left( {{\mathbb{R}^d}} \right)}$ stands for the powerset of ${{\mathbb{R}^d}}$.

Dirac's $\delta$ distributions are certainly welcome to "digest" those very high dimensional "integrals". However, we may also be interested in "probability distributions" like

$p\left[ {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right] \propto \sum\limits_{d = 1}^{ + \infty } {\int\limits_{\wp \left( {{\mathbb{R}^d}} \right)} {{\text{D}}\Gamma \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\int\limits_{{\mathbb{R}^ + }} {{\text{d}}\sigma {\sigma ^{ - M}}{e^{ - \sum\limits_{m = 1}^M {\frac{{{{\left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}}^2}}}{{2{\sigma ^2}}}} }}p\left( {\sigma ,z\left( 1 \right),f,g,\Gamma ,d} \right)} } } } } } $

Please, what can you say about those important "probability distributions" beyond the fact that they should better not be invariant by permutation of the time points, i.e. not finitely De Finetti-exchangeable, otherwise the chronological order, that is the time would be lost (conjecture)?

What can you say about such strange looking "functional integrals" (for the state-space and output equations $f$ and $g$) and even "set-theoretic integrals" (for phase space $\Gamma$) over sets having cardinal at least ${\beth_2}$? Are they already well-known in some branch of mathematics I do not know yet or are they only abstract nonsense?

Clearly, the noninformative case is the most important one. Hence, a definitive answer to my question could be something like this:

If $p\left( {z\left( 1 \right),f,g} \right)$ is the "improper non-informative prior probability distributions" over $\Gamma \times {\Gamma ^\Gamma } \times {\mathbb{R}^\Gamma }$ $p\left( {z\left( 1 \right),f,g} \right) \propto 1$ then the "marginal probability distribution"

$p\left[ {\left. {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right|\Gamma ,d} \right] = \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}} } } } $

is the improper uniform probability distribution over ${\mathbb{R}^M}$.

Your are free to restrict the sets of the state-space equations and output equations if necessary. However, contrary to what is suggested below, for a discrete-time dynamical system the state space equation $f$ needs not be continuous (the classical counterexample for phase space $\Gamma = \left[ {0,1} \right]$ is the Bernoulli shift that is discontinuous at $1/2$). Hence, a priori ${\beth_2}$ of them must be marginalized out.

Those beasts look very important to me because there are many problems of interest where we know a priori that the signal is deterministic but we don't known the underlying dynamical system (and output equation) and there is little hope to ever know it (generally speaking, dynamical systems identification is very delicate and difficult). Hence the deterministic model is unknown and not operationnal and by definition it is not legitimate to introduce a (objectively) stochastic one. So how to handle and process those signals properly? In theory, we perfectly know how to do it: just marginalize over all possible dynamical systems. But that does not make sense mathematically, so that it seems that we don't know how to model and process deterministic signals of unknown origin.

Thanks.

Answer 1

Classical Bayesian analysis rests on first chosing a prior measure $m$, either finite (proper) or infinite (improper), then deriving the posterior probability of an event $A$ conditional on observed $B$ as $Pr(A|B)=m(A\cap B)/m(B)$.

As I said in my first answer, this is possible for $[0,1]^{[0,1]}$ with the product Lebesgue measure, but not for $\mathbb R^{\mathbb R}$ because there is no such thing as "product Lebesgue measure" there, albeit an improper prior.

But we might consider a wider concept of prior, that of conditional probability space axiomatised by Alfred Renyi (1955): a family of ordinary probability spaces $(\Omega,\mathcal A,P_B)_{B\in \mathcal B}$ (where $P_B(A) $ represents $Pr(A|B)$), satisfying some compatibility condition.

In the case of $\Omega=\mathbb R^{\mathbb R}$, the family $\mathcal B$ of conditioning events could be that of products $\prod_{z\in\mathbb R}E_z$ where all $E_z$s have finite Lebesgue measure. For such a $B$ the proba $P_B$ will be defined as the product of normalised Lebesgue measures $\mathcal L(\cdot\cap E_z)/\mathcal L(E_z)$. The $\sigma$-algebra $\mathcal A$ is that of sets $\prod_{z\in\mathbb R}E_z$ where (contrary to members of $\mathcal B$) all but uncountably many $E_z=\mathbb R$.

While maybe (I don't know) conceptually new for Bayesian theory, I think this doesn't change much for what is searched here, a foundation for Bayesian processing of signals arising from a dynamical system: you still need restrictions to define probabilities...

Answer 2

On $[0,1]^{[0,1]}$ there is a prior distribution (even a "proper" one) that corresponds to the idea of "totally unknown": the product uniform measure. But you cannot do any meaningful Bayesian analysis with such a prior. You know $f$ maps $z_1$ to $z_2$ etc and the posterior distribution of $f(z)$ is still uniform on $[0,1]\cap\{z_2,\ldots,z_n\}^c$ if $z\notin\{z_1,\ldots,z_{n-1}\}$. As a result, the distribution of the infinite sequence of outputs $(z_i)_{i\in\mathbb N}$ is the product $\prod_{i\in\mathbb N}dz_i$ : they are i.i.d. uniform random variables.

To go Bayesian meaningfully, you need a (possibly improper) prior distribution on a set of maps $\Gamma\to\Gamma$ such that your unknown $f$ certainly belongs to it. It may be counting measure on a countable set of possible maps (example $f_n(x)=$ the fractional part of $nx$, on $\Gamma=[0,1]$) or a measure $d\lambda/\lambda$ for a set $\{f_\lambda,\lambda>0\}$. But with no restriction on the dynamical system, nothing reasonable can be done.

What can be done is to treat this as a problem of interpolation: construct $f_n$ satisfying $f_n(z_i)=z_{i+1}$, $i=1,\ldots,n-1$ (piecewise linear,spline,...), in such a way that $f_n$ converges to $f$ if the infinite sequence $(z_n)$ is dense. Moreover, this approach is Bayesian (Kimeldorf & Wahba, 1970), but with a prior such as Brownian (piecewise linear) or its primitive (cubic spline), not the "non-informative prior".