I am currently reading "Statistical and Inductive Inference by Minimum Message Length" by C.S. Wallace. In this, Wallace gives a fairly informal account of Bayesian Inference which, in the case everything is discrete, is basically as follows:

  1. We start with a prior probability distribution over a space of models of interest $\Theta$

  2. For each $\theta \in \Theta$, we also know a priori the likelihood $\mathbb{P}(x|\theta)$ of observing data $x$ if the model $\theta$ is "true"

  3. We observe some data $x$

  4. We update our prior distribution with the posterior one using Bayes' rule; that is, we set $$\mathbb{P}(\theta|x) = \frac{\mathbb{P}(x|\theta) \mathbb{P}(\theta)}{\mathbb{P}(x)}$$ for each $\theta \in \Theta$.

In the case that $\Theta$ is continuous, this process is roughly repeated but with density functions replacing $\mathbb{P}$ as appropriate.

Now, Bayes rule requires $\theta$ and $x$ be part of the same sample space. However, our prior distribution describes only models, and not data points. As such, it seems technically necessary to construct a new space which allows us to talk simultaneously about the probability of models and of measuring certain data. My question is: how do we do this in general (or how do we usually do it in practice)?

This question may be somewhat vague, so I have come up with a more concrete example of the sort of thing I want to do, which seems quite general and useful in practice. Suppose we have:

  • A probability space $$(\Theta, \mathcal{F}, \mathbb{P})$$ of models of interest (which the $\theta$'s reside in)
  • A measurable space $$(\Omega, \mathcal{G})$$ of data points (which the $x$'s reside in)
  • A mapping $\mathbb{P}(\cdot|\cdot) : \mathcal{G} \times \Theta \to [0, 1]$ such that $\mathbb{P}( \cdot | \theta)$ is a probability measure on $(\Omega, \mathcal{G})$ for each $\theta \in \Theta$. (This is our likelihood function.)

We want to use this to somehow come up with a probability $\mathbb{P}'$ on $$(\Theta \times \Omega, \mathcal{F} \times \mathcal{G})$$ (where $\mathcal{F} \times \mathcal{G}$ denotes the product $\sigma$-algebra) in a way that preserves the original behaviour of $\mathbb{P}$. I want to say that we should define $$\mathbb{P}'(A \times B) = \int_A \mathbb{P}(B | \theta) \,d\mathbb{P}(\theta),$$ where $A \in \mathcal{F}$ and $B \in \mathcal{G}$ (which adds the requirement that $\mathbb{P}(B | \cdot)$ be $\mathcal{F}$-measurable for each fixed $B \in \mathcal{G}$), but I see a problem in that not all events in $\mathcal{F} \times \mathcal{G}$ have the form $A \times B$.

Can this be done? Or is there a completely separate way to approach Bayesian inference which avoids all these difficulties?

  • 2
    $\begingroup$ Take a look at de Finetti representation theorems. They characterize in what sense probability distributions over sample spaces describing the data can be expressed as marginal distributions over a product space involving "parameters". $\endgroup$
    – R Hahn
    Nov 17 '14 at 17:49
  • 3
    $\begingroup$ Since sets of the form $A\times B$ form a $\pi$-system, what you wrote does indeed determine $\mathbb{P}'$ uniquely. As a matter of fact, this is how one defines product measures (which is just the special case in which $\mathbb{P}(B|\theta)$ does not depend on $\theta$). So the answer to the question is that yes, it can be done, and one does it in exactly the way you suggest. $\endgroup$ Dec 18 '14 at 13:35
  • $\begingroup$ I was wondering about exactly the same point a while ago (see mathoverflow.net/questions/226438/…) and the answer I came up with is: Yes, you do need to change the old / cook up a new probability space. In that sense Bayesians say that they model $X$ and $\Theta$ but in fact they ASSUME already that there is a joint distribution and that $f_{X|\Theta}=...$ (the function they want). $\endgroup$ Mar 22 '16 at 9:28
  • $\begingroup$ For example: they say that $X$ is normally distributed with parameter $\theta = (\mu, \sigma^2)$. They actually mean: We have no clue how $X$ is distributed (and we dont need that knowledge) but we assume that there is a common prob. space on which a RV $X$ and $\Theta$ live, they have a joint density, and $f_{X|\Theta}(x, \theta) = f_{X,\Theta}/f_\Theta = \text{const} \exp(-(x-\mu)^2/2\sigma^2)$. $\endgroup$ Mar 22 '16 at 9:31

I am not an expert, but here is how I think of this:

You can always take your sample space to be the real line $R$.

Let $\mu$ be the probability distribution for $\theta$, say with density function $f(\theta)$.

For each $\theta\in R$, let $\nu_\theta$ be the corresponding probability distribution with density function $g_\theta(x)$.

Let $\pi_1$ and $\pi_2$ be the usual projections from $R^2$ to $R$.

For $E\subset R^2$, let

$$\Pi(E)=\int_R \nu_\theta(\pi_2^{-1}(E))d\mu(\theta)$$

Make enough technical assumptions on the $\nu_\theta$ so that $\Pi$ is a probability measure on $R^2$. (It suffices, I think, for $\nu_\theta(B)$ to be a Borel function of $\theta$ whenever $B$ is a measurable subset of $R$.)

Check that $\Pi$ has density function $f(\theta)g_\theta(x)$. Think of this as the joint density for $(\theta,x)$.

Then the posterior density of $\theta$ given an observation $x$ is



As you said, the treatment is informally. Generally in Bayesian statistics you begin with a probability space Ω on which all the probabilities of interest - the joint distribution of models and data - live.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.