# What's the probability distribution of a deterministic signal or how to marginalize dynamical systems? (functional integrals in probability theory)

Because I still have no idea how it is possible for me to write down seemingly important equations ... that don't make any sense (at least for me) and because I haven't got any helpful comment so far, I'll be happy to offer a +100 bounty (that is, almost all my reputation!) not for a definitive anwser to this weird question but only for any serious, relevant feedback, thought, opinion or advice.

In many signal processing calculations, the prior probability distribution of the theoretical signal of interest (not the noisy experimental signal) is required. Here is the concrete problem from which my question has arisen:

Is there a Bayesian theory of deterministic signal? Prequel and motivation for my previous question

In random signal theory (à la Shannon), this probability distribution typically is a stochastic process, e.g. an i.i.d. stochastic process if you are a frequentist or an exchangeable stochastic process if you are a Bayesian, in the most basic cases.

What do such probability distributions become in deterministic signal theory/dynamical system theory?, that is the question.

To make it simple, consider a discrete-time real deterministic signal

$s\left( {1} \right),s\left( {2} \right),...,s\left( {M} \right)$

For instance, it can be obtained by sampling a continuous-time real deterministic signal.

By the standard definition of a discrete-time deterministic dynamical system, there exists:

• a phase space $\Gamma$, e.g. $\Gamma \subset \mathbb{R} {^d}$, $\Gamma = \left[ {0,1} \right]$, etc.

• a state-space equation $f:\Gamma \to \Gamma$ such as $z\left( {m + 1} \right) = f\left[ {z\left( m \right)} \right]$;

• an output or observation equation $g:\Gamma \to \mathbb{R}$ such as $s\left( m \right) = g\left[ {z\left( m \right)} \right]$;

• an initial condition $z\left( 1 \right)\in \Gamma$ in the domain of definition of $f$.

Hence, by definition we have

$\left[ {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right] = \left\{ {g\left[ {z\left( 1 \right)} \right],g\left[ {f\left( {z\left( 1 \right)} \right)} \right],...,g\left[ {{f^{M - 1}}\left( {z\left( 1 \right)} \right)} \right]} \right\}$

or, in probabilistic notations

$p\left[ {\left. {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right|z\left( 1 \right),f,g,\Gamma ,d} \right] = \prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}}$

Therefore, by "total probability and the product rule", the "marginal joint prior probability distribution" for a discrete-time deterministic signal conditional on phase space $\Gamma$ and its dimension $d$ formally/symbolically writes

$p\left[ {\left. {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right|\Gamma ,d} \right] = \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}p\left( {z\left( 1 \right),f,g} \right)} } } }$

Should phase space $\Gamma$ and its dimension $d$ be also unknown a priori, they should be marginalized as well so that the most general "marginal prior probability distribution" for a discrete-time deterministic signal I'm considering formally/symbolically writes

$p\left[ {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right] = \sum\limits_{d = 1}^{ + \infty } {\int\limits_{\wp \left( {{\mathbb{R}^d}} \right)} {{\text{D}}\Gamma \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}p\left( {z\left( 1 \right),f,g,\Gamma ,d} \right)} } } } } }$

where ${\wp \left( {{\mathbb{R}^d}} \right)}$ stands for the powerset of ${{\mathbb{R}^d}}$.

Dirac's $\delta$ distributions are certainly welcome to "digest" those very high dimensional "integrals". However, we may also be interested in "probability distributions" like

$p\left[ {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right] \propto \sum\limits_{d = 1}^{ + \infty } {\int\limits_{\wp \left( {{\mathbb{R}^d}} \right)} {{\text{D}}\Gamma \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\int\limits_{{\mathbb{R}^ + }} {{\text{d}}\sigma {\sigma ^{ - M}}{e^{ - \sum\limits_{m = 1}^M {\frac{{{{\left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}}^2}}}{{2{\sigma ^2}}}} }}p\left( {\sigma ,z\left( 1 \right),f,g,\Gamma ,d} \right)} } } } } }$

Please, what can you say about those important "probability distributions" beyond the fact that they should better not be invariant by permutation of the time points, i.e. not finitely De Finetti-exchangeable, otherwise the chronological order, that is the time would be lost (conjecture)?

What can you say about such strange looking "functional integrals" (for the state-space and output equations $f$ and $g$) and even "set-theoretic integrals" (for phase space $\Gamma$) over sets having cardinal at least ${\beth_2}$? Are they already well-known in some branch of mathematics I do not know yet or are they only abstract nonsense?

Clearly, the noninformative case is the most important one. Hence, a definitive answer to my question could be something like this:

If $p\left( {z\left( 1 \right),f,g} \right)$ is the "improper non-informative prior probability distributions" over $\Gamma \times {\Gamma ^\Gamma } \times {\mathbb{R}^\Gamma }$ $p\left( {z\left( 1 \right),f,g} \right) \propto 1$ then the "marginal probability distribution"

$p\left[ {\left. {s\left( 1 \right),s\left( 2 \right),...,s\left( M \right)} \right|\Gamma ,d} \right] = \int\limits_{{\mathbb{R}^\Gamma }} {{\text{D}}g\int\limits_{{\Gamma ^\Gamma }} {{\text{D}}f\int\limits_\Gamma {{{\text{d}}^d}z\left( 1 \right)\prod\limits_{m = 1}^M {\delta \left\{ {g\left[ {{f^{m - 1}}\left( {z\left( 1 \right)} \right)} \right] - s\left( m \right)} \right\}} } } }$

is the improper uniform probability distribution over ${\mathbb{R}^M}$.

Your are free to restrict the sets of the state-space equations and output equations if necessary. However, contrary to what is suggested below, for a discrete-time dynamical system the state space equation $f$ needs not be continuous (the classical counterexample for phase space $\Gamma = \left[ {0,1} \right]$ is the Bernoulli shift that is discontinuous at $1/2$). Hence, a priori ${\beth_2}$ of them must be marginalized out.

Those beasts look very important to me because there are many problems of interest where we know a priori that the signal is deterministic but we don't known the underlying dynamical system (and output equation) and there is little hope to ever know it (generally speaking, dynamical systems identification is very delicate and difficult). Hence the deterministic model is unknown and not operationnal and by definition it is not legitimate to introduce a (objectively) stochastic one. So how to handle and process those signals properly? In theory, we perfectly know how to do it: just marginalize over all possible dynamical systems. But that does not make sense mathematically, so that it seems that we don't know how to model and process deterministic signals of unknown origin.

Thanks.

• No idea at all (just like me)? – Fabrice Pautot Apr 12 '16 at 20:22
• Keep in mind that the notification system doesn't work if there's a space between @ and the user name. (Your last comment should have gotten through though.) – Todd Trimble Apr 23 '16 at 12:34
• 21 edits by OP. – Gerry Myerson Apr 28 '16 at 23:10
• @everyone. Same question on PO: physicsoverflow.org/35989/… including a nice discussion about the line of reasoning underlying this unusual question. – Fabrice Pautot Apr 29 '16 at 10:18
• the reason excessive edits are frowned upon, is that each edit promotes the question to the front page, so this is seen as an inappropriate way to attract attention to a question. – Carlo Beenakker Apr 29 '16 at 11:09

Classical Bayesian analysis rests on first chosing a prior measure $m$, either finite (proper) or infinite (improper), then deriving the posterior probability of an event $A$ conditional on observed $B$ as $Pr(A|B)=m(A\cap B)/m(B)$.

As I said in my first answer, this is possible for $[0,1]^{[0,1]}$ with the product Lebesgue measure, but not for $\mathbb R^{\mathbb R}$ because there is no such thing as "product Lebesgue measure" there, albeit an improper prior.

But we might consider a wider concept of prior, that of conditional probability space axiomatised by Alfred Renyi (1955): a family of ordinary probability spaces $(\Omega,\mathcal A,P_B)_{B\in \mathcal B}$ (where $P_B(A)$ represents $Pr(A|B)$), satisfying some compatibility condition.

In the case of $\Omega=\mathbb R^{\mathbb R}$, the family $\mathcal B$ of conditioning events could be that of products $\prod_{z\in\mathbb R}E_z$ where all $E_z$s have finite Lebesgue measure. For such a $B$ the proba $P_B$ will be defined as the product of normalised Lebesgue measures $\mathcal L(\cdot\cap E_z)/\mathcal L(E_z)$. The $\sigma$-algebra $\mathcal A$ is that of sets $\prod_{z\in\mathbb R}E_z$ where (contrary to members of $\mathcal B$) all but uncountably many $E_z=\mathbb R$.

While maybe (I don't know) conceptually new for Bayesian theory, I think this doesn't change much for what is searched here, a foundation for Bayesian processing of signals arising from a dynamical system: you still need restrictions to define probabilities...

• Thanks. As you know, exchangeability (infinite and finite) is indispensable for the Bayesian paradigm since, by the De Finetti representation theorem, it guarantees the existence of prior probability measures. But exchangeability is nothing but the absence of a chronological order or time arrow, i.e. the absence of determinism. Due to my lack of knowledge and understanding, I was not sure that our ignorance about the dynamical system would be sufficient to make the time arrow disappear. But it does so that the full probabilistic machinery still works, as it should. – Fabrice Pautot Mar 22 '18 at 16:15

On $[0,1]^{[0,1]}$ there is a prior distribution (even a "proper" one) that corresponds to the idea of "totally unknown": the product uniform measure. But you cannot do any meaningful Bayesian analysis with such a prior. You know $f$ maps $z_1$ to $z_2$ etc and the posterior distribution of $f(z)$ is still uniform on $[0,1]\cap\{z_2,\ldots,z_n\}^c$ if $z\notin\{z_1,\ldots,z_{n-1}\}$. As a result, the distribution of the infinite sequence of outputs $(z_i)_{i\in\mathbb N}$ is the product $\prod_{i\in\mathbb N}dz_i$ : they are i.i.d. uniform random variables.

To go Bayesian meaningfully, you need a (possibly improper) prior distribution on a set of maps $\Gamma\to\Gamma$ such that your unknown $f$ certainly belongs to it. It may be counting measure on a countable set of possible maps (example $f_n(x)=$ the fractional part of $nx$, on $\Gamma=[0,1]$) or a measure $d\lambda/\lambda$ for a set $\{f_\lambda,\lambda>0\}$. But with no restriction on the dynamical system, nothing reasonable can be done.

What can be done is to treat this as a problem of interpolation: construct $f_n$ satisfying $f_n(z_i)=z_{i+1}$, $i=1,\ldots,n-1$ (piecewise linear,spline,...), in such a way that $f_n$ converges to $f$ if the infinite sequence $(z_n)$ is dense. Moreover, this approach is Bayesian (Kimeldorf & Wahba, 1970), but with a prior such as Brownian (piecewise linear) or its primitive (cubic spline), not the "non-informative prior".

• Thanks Jean. Seems like you are the first one to understand or to seriously address my question. Any reference about the prior on [0,1]^[0,1] and Jeffreys' please? When you say that nothing can be done without restricting the set of all dynamical systems, can we prove that's impossible or is it just because we do not know? – Fabrice Pautot Mar 16 '18 at 20:04
• Nothing reasonable can be done to identify a "dynamical system" from a finite sample using the non-informative prior. It's not a theorem, but you can see the posterior distribution is useless. – Jean Duchon Mar 17 '18 at 8:36
• Merci beaucoup Jean. But I don't want to make any inference about the underlying dynamical system (for this purpose, I would follow the usual way: Ruelle-Takens-Sauer embedding theorem/delay method, symplectic techniques, surrogate data, etc.). I just want to supply a prior probability distribution for my signal in order to finish my signal processing calculations... – Fabrice Pautot Mar 17 '18 at 9:49
• If you tell me that we can properly define an uninformative unifrom product prior over the set of all state-space equations at least from [0,1] to [0,1], then the question boils down to: can I do the corresponding functional integrals in order to get the marginal distribution or, on the contrary, can we prove that's not possible? – Fabrice Pautot Mar 17 '18 at 9:55
• Besides, there is a much better Bayesian way for interpolation: the Poincaré-Jaynes-Bretthorst one. See this post: mathoverflow.net/questions/47675/… – Fabrice Pautot Mar 17 '18 at 15:23