Sanov's theorem and Dvoretzky–Kiefer–Wolfowitz's inequality tell us how fast the empirical distribution concentrates around the true underlying probabilty distribution.

What is known about the concentration of the posterior distribution ? Do such inequalities exist for a mode, a median, the mean or a sample of the posterior ?

Formally, let $\left\lbrace \mathbb{P}_{\theta} \ : \ \theta \in \Theta \right\rbrace$ be a set of probability distributions on $\mathbb{R}$ and $\mu_0$ a prior on $\Theta$.

Let $\mu_n$ be the posterior distribution, i.e. the regular conditional distribution of $\theta$ given $X_1,...,X_n$ when the distribution of $(\theta,X_1,...,X_n)$ is $\mathbb{P}=\mu_0 \otimes \mathbb{P}_{\theta}^{\otimes n}$. $$\mu_n(A,x_1,...,x_n) = \mathbb{P}(\theta \in A \ | \ X_1 = x_1,...,X_n = x_n)$$ If the model is true, this is indeed the distribution of $\theta$ when one has observed n i.i.d samples $(X_1,...,X_n)$ with values $(x_1,...,x_n)$. Even if the model is false, given $n$ values $(x_1,...,x_n)$, $A \mapsto \mu_n(A,x_1,...,x_n)$ still defines a probability distribution on $\Theta$.

Now if I'm a frequentist, I do not agree that $\theta$ is a random variable, I believe it is a parameter with a true (but unknown) value $\theta_{\text{true}}$ and the data $(X_1,...X_n)$ are then i.i.d with true underlying (but unknown) distribution $\mathbb{P}_{\theta_{\text{true}}}^{\otimes n}$. I then a get a random probability distribution $A \mapsto \mu_n(A,X_1,...,X_n)$ on $\Theta$ for which I can take a mode, a median (if $\Theta$ is a subset of $\mathbb{R})$), the mean (also if $\Theta$ is a subset of $\mathbb{R})$) or even a sample (which are all random quantities because they depend on the data).

I'm looking for finite-time upper bounds on the quantities $\mathbb{P}_{\theta_{\text{true}}}^{\otimes n}\left( \bullet \notin A \right)$ where $\bullet$ stands for the mean, a median, a mode or a sample of the posterior $\mu_n$ and $A$ is a measurable subset of $\Theta$ containing $\theta_{\text{true}}$.

I don't know much about bayesian statistics and would be happy with results in any setting ($\Theta$ can be as simple as you want, like a standard exponential family, or even the family of Bernoulli distributions, and $A$ can be any specific neighbourhood of $\theta_{\text{true}}$).

  • $\begingroup$ You might look into the work by Van der Vaart. $\endgroup$ – Daniel Roy Nov 21 '13 at 19:57
  • $\begingroup$ Can you define precisely what do you mean by posterior, ie define $\mu_n$? $\endgroup$ – ofer zeitouni Nov 22 '13 at 5:21
  • $\begingroup$ @Daniel Roy : Could you be a little more specific ? Maybe point out a paper or a chapter of a book ? :) $\endgroup$ – Adrien Nov 22 '13 at 11:25
  • $\begingroup$ @ofer zeitouni: please tell me if they are things which are still not clear. $\endgroup$ – Adrien Nov 22 '13 at 11:27
  • $\begingroup$ Saying that Bayesians think things like $\theta$ are "random variables" is something of a frequentist parody of the Bayesian view. If one goes by the standard definitions used by mathematicians since Kolmogorov's book came out about eight decades ago, however, it is correct. That makes a flaw in the standard definitions apparent. The mass of the planet Neptune does not vary randomly, yet Bayesians may assign a probability distribution to it. One solution to this unpleasantness is to banish the word "random" from the theory of probability, as famously proposed by.... $\endgroup$ – Michael Hardy Jan 21 '14 at 17:29

Well, in this case, the posterior distribution on $\Theta$, assuming $p_\theta(x)$ is a nice density of the $x_i$s, and assuming $\Theta\subset R$ and $\mu_0$ having density $p_0$ wrt Lebesgue, is $$\frac{p_0(\theta) e^{\sum_{i=1}^n g_\theta(x_i)}}{\int_\Theta p_0(\theta) e^{\sum_{i=1}^n g_\theta(x_i)} d\theta},$$ where $g_\theta(x)=-\log p_\theta(x)$. For $n$ large $$\sum_{i=1}^n g_\theta(x_i)=n \langle L_n, g_\theta(\cdot)\rangle= n \langle p_t,g_\theta(\cdot)\rangle+ \sqrt{n} G_\theta$$ where the error term $G_\theta$ converges to a Gaussian process and $p_t$ is your true distribution. If $\theta\to \langle p_t,g_\theta\rangle$ has a unique minimizer $\hat\theta\in \Theta$, the posterior measure thus will concentrate around $\hat\theta$. In particular, you can (from the estimates on convergence of $L_n$, which are exponential) estimate how fast the convergence toward $\hat \theta$ occur. Is that what you had in mind?

I am sure all this is written in standard references in the statistics literature, however I am no sure where.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.