7
$\begingroup$

Suppose a data generating process (DGP) is parameterized by some unknown parameter $\theta_0$, say $P_{\theta_0}$, and we want to estimate the value of $\theta_0$ using Bayesian method. Let $\pi(\theta)$ be the prior over the possible values of $\theta$, $\Theta$.

I understand that for almost all priors, if the data observed are iid, then the Bayesian posterior will eventually concentrate on $\theta_0$, as the number of observations ($n$) tends to infinity. (Correct me if I'm wrong.)

My question: Are there any results that predict differential rates of convergence of the posterior distribution based on different priors?

For example, suppose $\Theta=\{\theta_0,\theta_1\}$ and the DGP is $P_{\theta_0}$. Consider two priors on $\Theta$: $$\pi_1(\theta_0)=0.2,\qquad \pi_1(\theta_1)=0.8$$ and $$\pi_2(\theta_0)=\pi_2(\theta_1)=0.5.$$ Are there any results that says the posterior based on $\pi_2$ converges faster than that based on $\pi_1$? (or the other way around?)

Intuitively I would expect the posterior based on $\pi_2$ to have faster convergence, as it is "closer" to $\theta_0$. But in my experience, intuition is hardly reliable when it comes to probability theory.

Note: I've asked this quesiton on Math.SE but received no answer. I thought I would try my luck here.

$\endgroup$

1 Answer 1

4
$\begingroup$

One can measure the rate of convergence of the posterior distribution with density $p_n$ to the Dirac probability distribution at $\theta_0$ by how large the ratios $p_n(\theta_0)/p_n(\theta)$ are for $\theta\ne\theta_0$, where $\theta_0$ is the "true" value of the parameter. One has $$ \frac{p_n(\theta_0)}{p_n(\theta)}= \frac{\pi(\theta_0)}{\pi(\theta)}\,\frac{L_n(\theta_0)}{L_n(\theta)}, $$ where $\pi$ is the prior density and $L_n(\theta):=\prod_1^n f_\theta(X_i)$ is the likelihood value based on the first $n$ of the iid observations $X_1,X_2,\dots$ with common density $f_{\theta_0}$. It is clear from the above display that, for the same "data" $X_1,X_2,\dots$, the greater is the prior ratio $\pi(\theta_0)/\pi(\theta)$ the greater is the posterior "contrast" ratio $p_n(\theta_0)/p_n(\theta)$. In particular, in the extreme case when (say) $\pi(\theta)$ is $1$ if $\theta=\theta_0$ and is $0$ otherwise, the prior ratio is $\infty$ for any $\theta\ne\theta_0$, and hence so is the "contrast" ratio $p_n(\theta_0)/p_n(\theta)$.

One may want to measure the posterior contrast by one number, say by averaging the posterior contrast over the data space and over the parameter space, to get something like $$\int\mu(d\theta)\,\mathsf{E}\ln\frac{p_n(\theta_0)}{p_n(\theta)}= \int\mu(d\theta)\,\ln\frac{\pi(\theta_0)}{\pi(\theta)} +n\int\mu(d\theta)\,\mathsf{E}\ln\frac{f_{\theta_0}(X_1)}{f_{\theta}(X_1)},$$ where $\mu$ is a measure on the parameter space.
So, the greater is the average prior contrast $\int\mu(d\theta)\,\ln\frac{\pi(\theta_0)}{\pi(\theta)}$ the greater is the average posterior contrast $\int\mu(d\theta)\,\mathsf{E}\ln\frac{p_n(\theta_0)}{p_n(\theta)}$.

In particular, your prior $\pi_2$ should result, by this logic, in a greater posterior contrast than $\pi_1$.

Addendum: The average prior contrast and hence the average posterior contrast may behave somewhat surprisingly, though. E.g., suppose that the parameter space is $\Theta=\mathbb{R}$, $\theta_0=0$, $\pi$ is the normal density with mean $\theta_1$ and variance $\tau^2$, $f_\theta$ is the normal density with mean $\theta$ and variance $\sigma^2$, and $\mu$ is the normal distribution with mean $0$ and variance $\gamma^2$. Then for the average prior and posterior contrasts one has $$\int\mu(d\theta)\,\ln\frac{\pi(\theta_0)}{\pi(\theta)}=\frac{\gamma^2}{2\tau^2}\quad\text{and}\quad \int\mu(d\theta)\,\mathsf{E}\ln\frac{p_n(\theta_0)}{p_n(\theta)}= \frac{\gamma^2}2\Big(\frac1{\tau^2}+\frac n{\sigma^2}\Big),$$ respectively. As could be expected, both these average contrasts are the greater the smaller is the variance $\tau^2$ of the prior distribution. However, neither of these contrasts depends on the prior mean $\theta_1$ ! This will be so for any distribution $\mu$ symmetric about $0$ (or, more generally, about $\theta_0$) -- because, for the normal prior $\pi$ as in this addendum, the symmetrized prior contrast $$\ln\frac{\pi(0)}{\pi(\theta)}+\ln\frac{\pi(0)}{\pi(-\theta)} =\frac{\theta^2-2\theta \theta _1}{2 \tau ^2} +\frac{\theta^2+2\theta \theta _1}{2 \tau ^2} =\frac{\theta^2}{\tau^2}$$ does not depend on $\theta_1$.

$\endgroup$
1
  • $\begingroup$ Can you explain the relation between posterior contrast and sample complexity? $\endgroup$
    – user3180
    May 4, 2019 at 9:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.