# Probability spaces involved in using Bayesian Inference

I am currently reading "Statistical and Inductive Inference by Minimum Message Length" by C.S. Wallace. In this, Wallace gives a fairly informal account of Bayesian Inference which, in the case everything is discrete, is basically as follows:

1. We start with a prior probability distribution over a space of models of interest $\Theta$

2. For each $\theta \in \Theta$, we also know a priori the likelihood $\mathbb{P}(x|\theta)$ of observing data $x$ if the model $\theta$ is "true"

3. We observe some data $x$

4. We update our prior distribution with the posterior one using Bayes' rule; that is, we set $$\mathbb{P}(\theta|x) = \frac{\mathbb{P}(x|\theta) \mathbb{P}(\theta)}{\mathbb{P}(x)}$$ for each $\theta \in \Theta$.

In the case that $\Theta$ is continuous, this process is roughly repeated but with density functions replacing $\mathbb{P}$ as appropriate.

Now, Bayes rule requires $\theta$ and $x$ be part of the same sample space. However, our prior distribution describes only models, and not data points. As such, it seems technically necessary to construct a new space which allows us to talk simultaneously about the probability of models and of measuring certain data. My question is: how do we do this in general (or how do we usually do it in practice)?

This question may be somewhat vague, so I have come up with a more concrete example of the sort of thing I want to do, which seems quite general and useful in practice. Suppose we have:

• A probability space $$(\Theta, \mathcal{F}, \mathbb{P})$$ of models of interest (which the $\theta$'s reside in)
• A measurable space $$(\Omega, \mathcal{G})$$ of data points (which the $x$'s reside in)
• A mapping $\mathbb{P}(\cdot|\cdot) : \mathcal{G} \times \Theta \to [0, 1]$ such that $\mathbb{P}( \cdot | \theta)$ is a probability measure on $(\Omega, \mathcal{G})$ for each $\theta \in \Theta$. (This is our likelihood function.)

We want to use this to somehow come up with a probability $\mathbb{P}'$ on $$(\Theta \times \Omega, \mathcal{F} \times \mathcal{G})$$ (where $\mathcal{F} \times \mathcal{G}$ denotes the product $\sigma$-algebra) in a way that preserves the original behaviour of $\mathbb{P}$. I want to say that we should define $$\mathbb{P}'(A \times B) = \int_A \mathbb{P}(B | \theta) \,d\mathbb{P}(\theta),$$ where $A \in \mathcal{F}$ and $B \in \mathcal{G}$ (which adds the requirement that $\mathbb{P}(B | \cdot)$ be $\mathcal{F}$-measurable for each fixed $B \in \mathcal{G}$), but I see a problem in that not all events in $\mathcal{F} \times \mathcal{G}$ have the form $A \times B$.

Can this be done? Or is there a completely separate way to approach Bayesian inference which avoids all these difficulties?

• Take a look at de Finetti representation theorems. They characterize in what sense probability distributions over sample spaces describing the data can be expressed as marginal distributions over a product space involving "parameters". – R Hahn Nov 17 '14 at 17:49
• Since sets of the form $A\times B$ form a $\pi$-system, what you wrote does indeed determine $\mathbb{P}'$ uniquely. As a matter of fact, this is how one defines product measures (which is just the special case in which $\mathbb{P}(B|\theta)$ does not depend on $\theta$). So the answer to the question is that yes, it can be done, and one does it in exactly the way you suggest. – Martin Hairer Dec 18 '14 at 13:35
• I was wondering about exactly the same point a while ago (see mathoverflow.net/questions/226438/…) and the answer I came up with is: Yes, you do need to change the old / cook up a new probability space. In that sense Bayesians say that they model $X$ and $\Theta$ but in fact they ASSUME already that there is a joint distribution and that $f_{X|\Theta}=...$ (the function they want). – Fabian Werner Mar 22 '16 at 9:28
• For example: they say that $X$ is normally distributed with parameter $\theta = (\mu, \sigma^2)$. They actually mean: We have no clue how $X$ is distributed (and we dont need that knowledge) but we assume that there is a common prob. space on which a RV $X$ and $\Theta$ live, they have a joint density, and $f_{X|\Theta}(x, \theta) = f_{X,\Theta}/f_\Theta = \text{const} \exp(-(x-\mu)^2/2\sigma^2)$. – Fabian Werner Mar 22 '16 at 9:31

I am not an expert, but here is how I think of this:

You can always take your sample space to be the real line $R$.

Let $\mu$ be the probability distribution for $\theta$, say with density function $f(\theta)$.

For each $\theta\in R$, let $\nu_\theta$ be the corresponding probability distribution with density function $g_\theta(x)$.

Let $\pi_1$ and $\pi_2$ be the usual projections from $R^2$ to $R$.

For $E\subset R^2$, let

$$\Pi(E)=\int_R \nu_\theta(\pi_2^{-1}(E))d\mu(\theta)$$

Make enough technical assumptions on the $\nu_\theta$ so that $\Pi$ is a probability measure on $R^2$. (It suffices, I think, for $\nu_\theta(B)$ to be a Borel function of $\theta$ whenever $B$ is a measurable subset of $R$.)

Check that $\Pi$ has density function $f(\theta)g_\theta(x)$. Think of this as the joint density for $(\theta,x)$.

Then the posterior density of $\theta$ given an observation $x$ is

$$p_x(\theta)={f(\theta)g_\theta(x)\over\int_Rf(s)g_s(x)ds}$$

As you said, the treatment is informally. Generally in Bayesian statistics you begin with a probability space Ω on which all the probabilities of interest - the joint distribution of models and data - live.