Bayes statistics precisely formulated

Question

I am trying to learn something about Bayesian statistics, however, I am struggling already with the simplest equations and, moreover, with the very basic questions: What are we given? What is our goal?

As far as I understood the most basic setup we are given a probability space $(\Omega, \mathcal{A}, P)$, a space with $\sigma$-algebra $(V, \mathcal{B})$ and a random variable $X : \Omega \to V$ which has a density parametrized by $\lambda$, $f(x;\lambda)$ and we also assume that $\lambda$ itself is drawn from a random variable $\Lambda : \Omega \to \mathbb{R}$. Moreover we are given an observation $x$ and try to figure out something about $\lambda$ from this observation.

Then people usually write down something like $$p(\lambda|x) = \frac{p(x|\lambda) p(\lambda)}{p(x)}$$

The expressions above are then evaluated to

$$p(x|\lambda) = f(x;\lambda)$$ $$p(\lambda) = g(\lambda)$$

Question 1: What is the meaning of '$p$'? I can hardly believe that these are probabilities, because although one could give $P(X=x|\Lambda=\lambda)$ a useful meaning [as the factorization of the conditional expected function $E(1_{X = x} | \sigma(\Lambda))$], the symbol $p(\lambda|x) = P(\Lambda=\lambda|X=x)$ would somehow still be zero all the time (as $\Lambda$ is continuously valued). So I assume that these are supposed to be densities.

Question 2: What is $X$? Isnt it merely true that we have a function $\tilde{X} : \Omega \times \mathbb{R} \to \mathbb{R}$ [not even jointly measurable] with the property that for every $\lambda \in \mathbb{R}$ fixed, $X_\lambda := \tilde{X}(\cdot, \lambda)$ is a random variable with density $f(\cdot; \lambda)$? Or are we considering $X(\omega) = \tilde{X}(\omega, \Lambda(\omega))$? If so, why is this measurable?

Question 3: If $p(\cdot)$ is supposed to be a density, what random variables do we put inside $p(x|\lambda)$? I assume that its supposed to be the conditional density of $X_\lambda$ and $\Lambda$. But here is the question: can one define conditional density of two random variables without the two random variables being jointly distributed? I know that one can put $f_Y(y|X=x) = \frac{f_{(X, Y)}(x,y)}{f_X(x)}$ but are the two random variables $X_\lambda$ and $\Lambda$ jointly distributed? This seems like an unnatural assumption...

Question 4: I guess that even if one can define conditional densities of random variables even if they are not jointly distributed then $p(x|\lambda)$ is some kind of abstract definition, so why is that the simple term $f(x;\lambda)$?

I am sorry if these are very basic questions but all the resources I was able to find skip over all these details, so pointers are very welcome :-)

Answer 1

Disclaimer: this refers to the situation described in these lecture notes, where Bayes theorem is applied to events. It seems this is not what the OP has in mind, so I'm just leaving my answer for the record.

• What are we given? As described in the lecture notes (section 1.3), we start from the probability space $(\Omega,{\cal A},P)$. Both $x$ and $\lambda$ are events $\in{\cal A}$ and $p(x|\lambda)=p(x\cap \lambda)/p(\lambda)$ is the probability (not probability density) of event $x$ given event $\lambda$ (defined if $p(\lambda)\neq 0$). Bayes theorem then follows directly from $$p(x|\lambda)p(\lambda)=p(x\cap\lambda)=p(\lambda|x)p(x),$$ provided both $p(\lambda)$ and $p(x)$ are $\neq 0$.

The key to this precise formulation of Bayes theorem is to treat $x$ and $\lambda$ on the same footing. Both are events in ${\cal A}$, governed by the same probability $p$ of any event in ${\cal A}$ occuring, and the random variable $X:\Omega\rightarrow V$ plays the same role for both $x$ and $\lambda$. So $p(x|\lambda)$ is not some kind of abstract object, but directly obtained as the ratio of the probabilities that the joined event $x\cap\lambda$ happens and the probability that the event $\lambda$ happens. Both these events are in $\cal A$, so this ratio is well defined (assuming $p(\lambda)\neq 0$).

• What is our goal? To appreciate the usefulness of this equation, a simple but familiar example might help. You receive an email message in ALL-CAPS. What is the probability that it is spam? You have at your disposal a collection of spam messages from the past few years, and from that you can easily determine the fraction of those message written in capital letters. The fraction is $p(\lambda|x)$, with $x$ the event "I received a spam email" and $\lambda$ the event "the email I received contains only capital letters". If you know, or can estimate, what fraction $p(x)$ of your emails is spam and what fraction $p(\lambda)$ is in all-caps, then Bayes' formula tells you the probability $p(x|\lambda)$ that a message in all-caps is spam. This is how spam filters work.

Answer 2

Let $(\tilde{\Omega}, \tilde{\mathcal{A}})$ be a space with $\sigma$-algebra and let $(V, \mathcal{B}, P_V), (W, \mathcal{C}, P_W)$ be probability spaces. Let further $f : V \times W \to \mathbb{R}$ be in $L^1(V \times W)$ where $V \times W$ is endowed with the product sigma algebra $\mathcal{B} \otimes \mathcal{C}$ and let $g : W \to \mathbb{R}$ be in $L^1(W)$. Assume further that $f , g > 0$ (i.e. for every $x \in V, \theta \in W, f(x,\theta) > 0$ and $g(\theta) > 0$) and that for every fixed $\theta \in W$, $\int_V f(v, \theta) d\theta = 1$ (i.e. $f(\cdot, \theta)$ is a density) and that $\int_W g(\theta) d\theta = 1$ (i.e. $g$ is a density).

Let $X : \tilde{\Omega} \to V,~~ \Theta : \tilde{\Omega} \to W$ be measurable maps. We consider the (measurable!) map $(X, \Theta) : \tilde{\Omega} \to V \times W,~~ (X, \Theta)(\omega) = (X(\omega), \Theta(\omega))$. We put $\Omega := \tilde{\Omega}, \mathcal{A} := \sigma(X \times \Theta) := (X \times \Theta)^{-1}(\mathcal{B} \otimes \mathcal{C})$ and define a probability measure on $(\Omega, \mathcal{A})$ by $$P((X \times \Theta)^{-1}(U)) = \int_U f(x, \theta) g(\theta) d(x,\theta)$$ Notice that this definition makes sense as $(x, \theta) \mapsto f(x, \theta)g(\theta) \in L^1(V \times W)$ by Fubini!

Now we have the final probability space $(\Omega, \mathcal{A}, P)$ and a jointly distributed random variable $(X, \Theta) : \Omega \to (V \times W)$. By means of the projections $\pi_1 : V \times W \to V, \pi_1(v, w) = v$ and $\pi_2 : V \times W, \pi_2(v, w) = w$ we obtain back the measurable maps we started with as $X = \pi_1 \circ (X, \Theta) ~\text{and}~ \Theta = \pi_2 \circ (X, \Theta)$. Since we endowed $\Omega$ with a probability measure we can now really say that $X, \Theta$ are random variables. Observe that we easily verify that $\Theta$ has density $g$ (For every set $C \in C$ we have \begin{align*} P[\Theta \in C] &= P[\pi_2 \circ (X, \Theta) \in C] \\ &= P[(X, \Theta) \in V \times C] \\ &= \int_{V \times C} f(x,\theta)g(\theta) d(x,\theta) \\ &= \int_C \int_V f(x,\theta) dx g(\theta) d\theta \\ &= \int_C 1\cdot g(\theta) d\theta \end{align*} ) but it is completely unclear what density $X$ has, $X$ is just some random variable with 'unknown' distribution. Well, it is absolutely not unknown as $(X, \Theta)$ are jointly distributed but the density of $X$ is not interesting for us: Let us be very concrete here and choose fixed real numbers $\alpha, \beta$, let $\Theta$ be Beta$(\alpha, \beta)$ distributed and let $X$ be Bernoulli$(\theta)$ distributed. Encoded, this means that \begin{align*} V &=\{0,1\},\\ W &= (0,1), \\ f(x,\theta) &= \theta^x(1-\theta)^{1-x}, \\ \tilde{g}(\theta) &= \text{Beta}(\theta|\alpha, \beta) = \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} \theta^{\alpha - 1} (1-\theta)^{\beta-1}, \\ g(\theta) &= \frac{1}{\int_{(0,1)} \tilde{g}(\theta)}\tilde{g}(\theta) \end{align*} (the nasty constant will be just as uninteresting as the other ones, see below). Put $d := p(x) = f_X(x)$ then after doing the computation $$p(\lambda|x) = \frac{p(x|\lambda)p(\lambda)}{p(x)} = \text{const.} \theta^{x+\alpha-1} (1-\theta)^{\beta-x}$$ i.e. we know that $p(\lambda|x)$ is a density and we know that (up to a constant) it is a Beta distribution again! Since two densities that coincide up to a constant must be the same, it must be the (renormalized to the interval $(0,1)$) Beta distribution (somewhat independent of what the actual value of $f_X(x)$ was)!

As $(X, \Theta)$ are jointly distributed, there is a conditional density $f_{X|\Theta}$ given by \begin{align*} f_{X|\Theta}(x,\theta) &= \frac{f_{(X, \theta)}(x, \theta)}{f_\Theta(\theta)} \\ &= \frac{h(x, \theta)}{f_\Theta(\theta)} \\ &= \frac{f(x, \theta) g(\theta)}{g(\theta)} \\ &= f(x, \theta) \end{align*} This is the key point of confusion:

Although the Bayesian people claim that one models the density of $X$ somehow, what they actually mean is that they model the conditional density $f_{X|\Theta} = f$ and 'computing' the density $p(x|\lambda)$ does not actually mean that they compute something but that they compute beforehand that the conditional density is just $f$ and insert the definition of $f$!