What is the conjugate prior distribution of the Dirichlet distribution?

$\begingroup$ en.wikipedia.org/wiki/Dirichlet_distribution $\endgroup$– Steve HuntsmanApr 5, 2010 at 18:40

2$\begingroup$ ...does not list the conjugate prior. $\endgroup$– NeilApr 5, 2010 at 18:48

2$\begingroup$ There is some discussion here: stat.columbia.edu/~cook/movabletype/archives/2009/04/… $\endgroup$– Douglas ZareApr 5, 2010 at 18:59

$\begingroup$ This might be better. arxiv.org/abs/condmat/0603120 $\endgroup$– Steve HuntsmanApr 5, 2010 at 19:14
1 Answer
Neil sent me an email asking:
===
I read your post at http://www.stat.columbia.edu/~cook/movabletype/archives/2009/04/conjugate_prior.html and I was wondering if you could expand on how to update the Dirichlet conjugate prior that you provided in your paper:
S. Lefkimmiatis, P. Maragos, and G. Papandreou, Bayesian Inference on Multiscale Models for Poisson Intensity Estimation: Applications to PhotonLimited Image Denoising, IEEE Transactions on Image Processing, vol. 18, no. 8, pp. 17241741, Aug. 2009
In other words, given in your paper's notation the prior hyperparameters (vector $\mathbf{v}$, and scalar $\eta$), and $N$ Dirichlet observations (vectors $\mathbf{\theta}_n, n=1,\dots,N$), how do you update $\mathbf{v}$ and $\eta$?
===
Here is my response:
Conjugate pairs are so convenient because there is a standard and simple way to incorporate new data by just modifying the parameters of the prior density. One just multiplies the likelihood with its conjugate prior; the result has the same parametric form as the prior, and the new parameters can be readily "readoff" by comparing the likelihoodprior product with the prior parametric form. This is described in detail in all standard texts in Bayesian statistics such as Gelman et al. (2003) or Bernardo and Smith (2000).
In the case of the Dirichlet and its conjugate prior described in our paper and using its notation, after observing $N$ Dirichlet vectors $\mathbf{\theta}_n$, $n=1,\dots,N$, where each vector $\mathbf{\theta}_n$ is $D$ dimensional with elements $\theta_n[t]$, $t=1,\dots,D$, the $D+1$ hyperparameters should be updated as follows:
 $\eta_N = \eta_0 + N$
 $v_N[t] = v_0[t]  \sum_{n=1}^N \ln \theta_n[t], \quad t=1,\dots,D$, where $\eta_0$, $\mathbf{v}_0$ and $\eta_N$, $\mathbf{v}_N$ are the initial and updated model parameters, respectively.
You can verify this in a few lines of equations by following the previously described general rule.
Hope this helps!

2$\begingroup$ Thank you very much for your detailed answer! Am I right that "theta" represents categorical parameter vector observations (i.e., their components are in [0,1])? If the observations are "alpha", the parameter vectors of Dirichlet observations, then the update of v needs to be: v_N[j] = v_0[j] + sum_n [ digamma(sum_i alpha_n[i])  digamma(alpha_n[j]) ] because that's the expected log of each component value, right? Thanks again. This was very helpful. $\endgroup$– NeilApr 10, 2010 at 2:25

1$\begingroup$ Hi, at Level1 we have $p(\mathbf{\theta}\mathbf{\alpha}$, where $\theta$ is the Dirichlet observation lying on the $D$dimensional simplex (components between 0 and 1 whose sum is 1), and $\mathbf{\alpha}$ is the Dirichlet distribution parameter vector. At Level2 we have the conj. prior $p(\mathbf{\alpha}\eta,\mathbf{v})$ of our paper. In this setup, one observes $\theta$ vectors and updates the hyperparameters $\eta,\mathbf{v}$ that describe the density of $\mathbf{\alpha}$, which itself is considered hidden (it thus makes no sense to measure it). Hope it is clearer now! George $\endgroup$ Apr 14, 2010 at 3:17

1$\begingroup$ So to actually answer the question, is the Dirichlet distribution its own conjugate prior? $\endgroup$– isarandiOct 21, 2014 at 0:48

1$\begingroup$ Though the books on bayesian statistics are no doubt a better source, Wikipedia has a useful article on <conjugate priors for exponential family distributions> (such as the Dirichlet). $\endgroup$– drevickoMay 24, 2015 at 13:17

3$\begingroup$ @GeorgePapandreou: there is a typo in the reply. The actualisation should be$$v_N[t] = v_0[t] + \sum_{n=1}^N \ln \theta_n[t]$$not$$v_N[t] = v_0[t]  \sum_{n=1}^N \ln \theta_n[t]$$This confused a reader on Cross validated. $\endgroup$ Nov 25, 2015 at 17:39