Sanov's theorem and Dvoretzky–Kiefer–Wolfowitz's inequality tell us how fast the empirical distribution concentrates around the true underlying probabilty distribution.

What is known about the concentration of the posterior distribution ? Do such inequalities exist for a mode, a median, the mean or a sample of the posterior ?

Formally, let $\left\lbrace \mathbb{P}_{\theta} \ : \ \theta \in \Theta \right\rbrace$ be a set of probability distributions on $\mathbb{R}$ and $\mu_0$ a prior on $\Theta$.

Let $\mu_n$ be the posterior distribution, i.e. the regular conditional distribution of $\theta$ given $X_1,...,X_n$ when the distribution of $(\theta,X_1,...,X_n)$ is $\mathbb{P}=\mu_0 \otimes \mathbb{P}_{\theta}^{\otimes n}$. $$\mu_n(A,x_1,...,x_n) = \mathbb{P}(\theta \in A \ | \ X_1 = x_1,...,X_n = x_n)$$ If the model is true, this is indeed the distribution of $\theta$ when one has observed n i.i.d samples $(X_1,...,X_n)$ with values $(x_1,...,x_n)$. Even if the model is false, given $n$ values $(x_1,...,x_n)$, $A \mapsto \mu_n(A,x_1,...,x_n)$ still defines a probability distribution on $\Theta$.

Now if I'm a frequentist, I do not agree that $\theta$ is a random variable, I believe it is a parameter with a true (but unknown) value $\theta_{\text{true}}$ and the data $(X_1,...X_n)$ are then i.i.d with true underlying (but unknown) distribution $\mathbb{P}_{\theta_{\text{true}}}^{\otimes n}$. I then a get a random probability distribution $A \mapsto \mu_n(A,X_1,...,X_n)$ on $\Theta$ for which I can take a mode, a median (if $\Theta$ is a subset of $\mathbb{R})$), the mean (also if $\Theta$ is a subset of $\mathbb{R})$) or even a sample (which are all random quantities because they depend on the data).

I'm looking for finite-time upper bounds on the quantities $\mathbb{P}_{\theta_{\text{true}}}^{\otimes n}\left( \bullet \notin A \right)$ where $\bullet$ stands for the mean, a median, a mode or a sample of the posterior $\mu_n$ and $A$ is a measurable subset of $\Theta$ containing $\theta_{\text{true}}$.

I don't know much about bayesian statistics and would be happy with results in any setting ($\Theta$ can be as simple as you want, like a standard exponential family, or even the family of Bernoulli distributions, and $A$ can be any specific neighbourhood of $\theta_{\text{true}}$).

somecontexts in which I think it ought to be done. Those are applications in which one might be speaking of things like the mass of Neptune, and also contexts where the nature of Bayesianism and frequentism are discussed. $\endgroup$ – Michael Hardy Jan 21 '14 at 17:33truedifferences between Bayesianism and frequentism become far clearer if one's language is more accurate than what is now conventional. Bayesianism is the degree-of-belief interpretation of probability. $\endgroup$ – Michael Hardy Jan 21 '14 at 17:374more comments