what can be said about the choice of a prior in Bayesian statistics? - MathOverflow

what can be said about the choice of a prior in Bayesian statistics?

2011-02-17T20:58:22Z

When reading about the Bayesian approach to statistics, priors are an important component of the whole methodology.

Yet, it seems like priors are chosen without any specific theoretical motivation. There is the theory of conjugate priors, which is motivated mostly computationally, I believe, but still, I haven't seen a result in the line of "the choice of a certain prior will lead to faster convergence rate" or something similar to that.

Is there a good reference that analyzes the choice of a prior somehow, instead of always assuming that it is given, and assuming that it is completely the modeler's choice?

Answer by passerby51 for what can be said about the choice of a prior in Bayesian statistics?

2011-02-17T21:04:30Z

Short answer from someone who doesn't know much about Bayesian statistics:

You should read about "reference priors". Have a look here: http://arxiv.org/pdf/0904.0156

Answer by John D. Cook for what can be said about the choice of a prior in Bayesian statistics?

2011-02-17T23:18:17Z

There are many approaches to this problem. Here are three.

The subjective Bayes approach says the prior should simply quantify what is known or believed before the experiment takes place. Period. End of discussion.
The empirical Bayes approach says you can estimate your prior from the data itself. (In that case your "prior" isn't prior at all.)
The objective Bayes approach says to pick priors based on mathematical properties, such as "reference" priors that in some sense maximize information gain. Jim Berger gives a good defense of objective Bayes here.

In practice someone may use any and all of these approaches, even within the same model. For example, they may use a subjective prior on parameters where there is a considerable amount of prior knowledge and use a reference prior on other parameters that are less important or less understood.

Often it simply doesn't matter much what prior you use. For example, you might show that a variety of priors, say an optimistic prior and a pessimistic prior, lead to essentially the same conclusion. This is particularly the case when there's a lot of data: the impact of the prior fades as data accrue. But for other applications, such as hypothesis testing, priors matter more.

Answer by none for what can be said about the choice of a prior in Bayesian statistics?

2011-02-18T03:59:18Z

Sometimes you have to pick the prior from a probability distribution, called a hyperprior. That itself can be drawn from a hyperhyperprior and so forth. Eventually you to just pick something: see Sunrise problem.

Answer by Steven Landsburg for what can be said about the choice of a prior in Bayesian statistics?

2011-02-19T04:54:53Z

You can (at least in spirit, modulo technical details) define a prior by assigning to each hypothesis $H$ the probability $2^{-K(H)}$, where $K$ is Kolmogorov complexity. Obviously this prior is not as good as the prior that assigns to each hypothesis its true probability, but there are multiple senses in which it's (provably) almost as good. There are, for example, theorems saying that when you use this method to predict the values of a binary string, the total expected prediction error over infinitely many predictions is bounded above by a (finite) constant. An important key phrase here is minimum message length.

Answer by G. Grothendieck for what can be said about the choice of a prior in Bayesian statistics?

2011-02-20T16:26:06Z

The key point is to show that your analysis does not depend much on the prior in the first place.

You could try

a conjugate prior (if one exists)
a (possibly improper) uniform prior and
a Jeffreys prior (justified by its invariance under re-parameterization)

and if you get answers that are reasonably close then that should provide some support that your analysis does not depend on arbitrary choice of prior.

BUGS and similar systems give quite a bit of flexibility regarding the prior but otherwise, from a practical viewpoint, easy computability and the availability of software may restrict your choice of prior.

See: