2
$\begingroup$

Sanov's theorem and Dvoretzky–Kiefer–Wolfowitz's inequality tell us how fast the empirical distribution concentrates around the true underlying probabilty distribution.

What is known about the concentration of the posterior distribution ? Do such inequalities exist for a mode, a median, the mean or a sample of the posterior ?

Formally, let $\left\lbrace \mathbb{P}_{\theta} \ : \ \theta \in \Theta \right\rbrace$ be a set of probability distributions on $\mathbb{R}$ and $\mu_0$ a prior on $\Theta$.

Let $\mu_n$ be the posterior distribution, i.e. the regular conditional distribution of $\theta$ given $X_1,...,X_n$ when the distribution of $(\theta,X_1,...,X_n)$ is $\mathbb{P}=\mu_0 \otimes \mathbb{P}_{\theta}^{\otimes n}$. $$\mu_n(A,x_1,...,x_n) = \mathbb{P}(\theta \in A \ | \ X_1 = x_1,...,X_n = x_n)$$ If the model is true, this is indeed the distribution of $\theta$ when one has observed n i.i.d samples $(X_1,...,X_n)$ with values $(x_1,...,x_n)$. Even if the model is false, given $n$ values $(x_1,...,x_n)$, $A \mapsto \mu_n(A,x_1,...,x_n)$ still defines a probability distribution on $\Theta$.

Now if I'm a frequentist, I do not agree that $\theta$ is a random variable, I believe it is a parameter with a true (but unknown) value $\theta_{\text{true}}$ and the data $(X_1,...X_n)$ are then i.i.d with true underlying (but unknown) distribution $\mathbb{P}_{\theta_{\text{true}}}^{\otimes n}$. I then a get a random probability distribution $A \mapsto \mu_n(A,X_1,...,X_n)$ on $\Theta$ for which I can take a mode, a median (if $\Theta$ is a subset of $\mathbb{R})$), the mean (also if $\Theta$ is a subset of $\mathbb{R})$) or even a sample (which are all random quantities because they depend on the data).

I'm looking for finite-time upper bounds on the quantities $\mathbb{P}_{\theta_{\text{true}}}^{\otimes n}\left( \bullet \notin A \right)$ where $\bullet$ stands for the mean, a median, a mode or a sample of the posterior $\mu_n$ and $A$ is a measurable subset of $\Theta$ containing $\theta_{\text{true}}$.

I don't know much about bayesian statistics and would be happy with results in any setting ($\Theta$ can be as simple as you want, like a standard exponential family, or even the family of Bernoulli distributions, and $A$ can be any specific neighbourhood of $\theta_{\text{true}}$).

| cite | improve this question | | | | |
$\endgroup$
  • 1
    $\begingroup$ Saying that Bayesians think things like $\theta$ are "random variables" is something of a frequentist parody of the Bayesian view. If one goes by the standard definitions used by mathematicians since Kolmogorov's book came out about eight decades ago, however, it is correct. That makes a flaw in the standard definitions apparent. The mass of the planet Neptune does not vary randomly, yet Bayesians may assign a probability distribution to it. One solution to this unpleasantness is to banish the word "random" from the theory of probability, as famously proposed by.... $\endgroup$ – Michael Hardy Jan 21 '14 at 17:29
  • 1
    $\begingroup$ ...Edwin Jaynes. To speak of "uncertain quanitities" instead of random variables will not become universally standard in probability theory before next Monday, if indeed it ever happens. However, there are some contexts in which I think it ought to be done. Those are applications in which one might be speaking of things like the mass of Neptune, and also contexts where the nature of Bayesianism and frequentism are discussed. $\endgroup$ – Michael Hardy Jan 21 '14 at 17:33
  • 1
    $\begingroup$ ....and actually, I think it's better to speak of things like a "population mean" rather than a "true" mean. The use of the word "true" in that way seems like an undue concession to the frequentist viewpoint. The true differences between Bayesianism and frequentism become far clearer if one's language is more accurate than what is now conventional. Bayesianism is the degree-of-belief interpretation of probability. $\endgroup$ – Michael Hardy Jan 21 '14 at 17:37
  • 1
    $\begingroup$ Focusing more on this question: The Bayesian also does not think $\theta$ is a random variable, and like the frequentist, believes it has a true but unknown value. However, according to Bayesianism, assigning a probability distribution to express the uncertainty about $\theta$ is the right way to do it. The Bayesian treats $\theta$ as the kind of thing that the Kolmogorovs of this world call "random variables", but that shows why one shouldn't take Kolmogorov's use of language too seriously in this context. I think it's unfortunate that some of the most illustrious.... $\endgroup$ – Michael Hardy Jan 21 '14 at 17:42
  • 1
    $\begingroup$ ....Bayesians, notably Duke University's Jim Berger, don't really care about things like this. Accuracy in language really does make things clearer. $\endgroup$ – Michael Hardy Jan 21 '14 at 17:43
2
$\begingroup$

Well, in this case, the posterior distribution on $\Theta$, assuming $p_\theta(x)$ is a nice density of the $x_i$s, and assuming $\Theta\subset R$ and $\mu_0$ having density $p_0$ wrt Lebesgue, is $$\frac{p_0(\theta) e^{\sum_{i=1}^n g_\theta(x_i)}}{\int_\Theta p_0(\theta) e^{\sum_{i=1}^n g_\theta(x_i)} d\theta},$$ where $g_\theta(x)=-\log p_\theta(x)$. For $n$ large $$\sum_{i=1}^n g_\theta(x_i)=n \langle L_n, g_\theta(\cdot)\rangle= n \langle p_t,g_\theta(\cdot)\rangle+ \sqrt{n} G_\theta$$ where the error term $G_\theta$ converges to a Gaussian process and $p_t$ is your true distribution. If $\theta\to \langle p_t,g_\theta\rangle$ has a unique minimizer $\hat\theta\in \Theta$, the posterior measure thus will concentrate around $\hat\theta$. In particular, you can (from the estimates on convergence of $L_n$, which are exponential) estimate how fast the convergence toward $\hat \theta$ occur. Is that what you had in mind?

I am sure all this is written in standard references in the statistics literature, however I am no sure where.

| cite | improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.