We can sample a discrete value from the multinomial distribution.

We can also sample the parameters of the multinomial distribution from its conjugate prior the dirichlet distribution.

Since the dirichlet distribution is part of the exponential family, it too must have a conjugate prior distribution in the exponential family.

I hope you see where I'm going: what happens as this chain of priors is taken to infinity?

For a simpler example, what happens with the self-conjugate Gaussian distribution?


Let's say that you have a distribution $F$ in the exponential family with density \begin{align} \newcommand{\mbx}{\mathbf x} \newcommand{\btheta}{\boldsymbol{\theta}} f(\mbx \mid \btheta) &= \exp\bigl(\eta(\btheta) \cdot T(\mbx) - g(\btheta) + h(\mbx)\bigr) \end{align}

Given independent realizations $\{x_1, x_2, \dotsc, x_n\}$ of $F$ (with unknown parameter $\theta$), then the distribution over $\theta$, $F'$, is the conjugate prior of $F$. The density of $F'$ is \begin{align} f(\btheta \mid \boldsymbol\phi) = L(\btheta \mid \mbx_1, \dotsc, \mbx_n) &= f(\mbx_1, \dotsc, \mbx_n \mid \btheta) \\\\ &\propto \prod_i f(\mbx_i\mid \btheta) \\\\ &= \textstyle\prod_i\exp\Bigl(\eta(\btheta) \cdot \textstyle T\left(\mbx_i\right) - g(\btheta) + h(\mbx_i)\Bigr) \\\\ &\propto \textstyle\prod_i\exp\Bigl(\eta(\btheta) \cdot \textstyle T\left(\mbx_i\right) - g(\btheta)\Bigr) \\\\ &= \textstyle\exp\Bigl(\eta(\btheta) \cdot \bigl(\textstyle\sum_iT\left(\mbx_i\right)\bigr) - ng(\btheta)\Bigr) \\\\ &= \exp\bigl(\eta'(\boldsymbol \phi) \cdot T'(\btheta)\bigr) \end{align} where \begin{align} \eta'(\boldsymbol\phi) &= \begin{bmatrix} \sum_iT_1(\mbx_i) \\\\ \vdots \\\\ \sum_iT_k(\mbx_i) \\\\ \sum_i1 \end{bmatrix} & T'(\btheta) &= \begin{bmatrix} \eta_1(\btheta) \\\\ \vdots \\\\ \eta_k(\btheta) \\\\ -g(\btheta) \end{bmatrix}. \end{align} Thus, $F'$ is also in the exponential family ($T'$ replaced $\eta$ and $\eta'$ replaced $T$ since this distribution is over $\theta$ the parameter of the distribution over $x$.)

Interestingly, $\boldsymbol\phi$ has exactly one more parameter than $\btheta$ except in the rare case where natural parameter $\phi_{k+1}$ is redundant, but such a distribution would be very weird (it would mean that the number of observations $\mbx$, that is, $n$, tells you nothing about $\btheta$.)

So, to answer your question, with each conjugate prior you get exactly one more hyperparameter.

There are many conjugate priors of the Gaussian distribution depending on how you look at it. In my opinion, the analogy to the Multinomial-Dirichlet example would set things up as follows: assume that $n$ real-valued numbers are generated by a Gaussian with unknown mean and variance. Then, the distribution of the mean and variance given the data points is a three-parameter conjugate prior distribution whose sufficient statistics are the total of the samples, the total of the squares of the samples, and the number of samples.

  • $\begingroup$ Could you give an example of a conjugate prior for the beta distribution. My probability background isn't the greatest.... $\endgroup$ – user130327 Feb 21 '14 at 2:38
  • $\begingroup$ @user130327: If you write out the beta distribution in the above form and follow the steps, you can derive it yourself. You should end up with the same answer as is found here: mathoverflow.net/questions/20399/…. If you like, I can derive it for you. $\endgroup$ – Neil Feb 21 '14 at 4:39
  • $\begingroup$ If you don't mind it would be appreciated. Thank you. $\endgroup$ – user130327 Feb 21 '14 at 20:31
  • $\begingroup$ I have tried to do what you suggested but I don't seem to be getting anywhere. I am new to this probability area (1 week and a half). I was really hoping you could show me the derivation Neil. Thank you very much. Josh $\endgroup$ – user130327 Feb 27 '14 at 15:17
  • $\begingroup$ @user130327: Do you want to ask a question here or on math.stackexchange or stats.stackexchange? I'll have some time on Monday. $\endgroup$ – Neil Mar 1 '14 at 17:06

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.