# convergence of Bayesian posterior with non iid data

Let $(\epsilon_t)_t$ be a sequence of iid random variables, distributed according to the density $f:\mathbb{R}\to (0,\infty)$ and $$x_t = q( \theta^\star, x_1,x_2, \ldots, x_{t-1}) + \epsilon_t \,.$$ Assume that the derivative of $q$ with respect to $\theta$ is bounded from below, i.e. for all $\theta, x$ $$\frac{\partial q(\theta, (x_r)_{r<t})}{\partial \theta} \geq c > 0\,.$$ The posterior likelihood is defined as $$L_t(\theta) = \sum_{s=1}^t \log f \Big(x_s-q(\theta, (x_r)_{r<t})\Big) + \log \pi_0(\theta)\,,$$ where the prior density $\pi_0:\mathbb{R} \to (0,\infty)$ is strictly positive.

Question: Are those assumptions sufficient to conclude that the posterior $$\pi_t(\theta) = \frac{e^{L_t(\theta)}}{\int e^{L_t(z)} dz}$$ converges almost surely to a dirac measure on $\theta^\star$? How could I show this? What would be a good reference?

I know that Bernstein von-Mises Theorem yields this type of result, but I could not find a version which is general enough to cover this case.

Thanks a lot!

## 1 Answer

Probably not true just with the assumptions you have there. It seems to accommodate a model in which $\theta$ is the starting point of a random walk. That would only require $q = \theta + x_1 + ... + x_t$. It is known and easy to believe that in this case you cannot estimate $\theta$ consistently, and I would think that therefore the posterior should not collapse to a point mass. You can calculate the posterior explicitly in case $\theta, \epsilon_t$ are i.i.d. normal see that that is true.

• Do you know a reference arguing that $\theta$ can not be estimated consistently in this case? Thank you. – Peter Nov 14 '16 at 22:26
• Do you know of a restriction on $q$ that ensures convergence (allowing for dependence $x_1, \ldots, x_{t-1}$)? – Peter Nov 14 '16 at 23:31
• I regret that I don't know a reference, but the technique is to show that the measures starting from different $\theta$ are absolutely comntinuous up to and including infinity. Then, you can't have a function that is $\theta_1$ with probability 1 under $P_{\theta_1}$ and $\theta_2$ under $P_{\theta_2}$. I do not know any general conditions in this framework, but is there a specific problem you are working on ? – user83457 Nov 15 '16 at 8:55
• Thanks again Michael! I am interested in the case where $q(\theta,m(L_{t-1}))$ is a function of $\theta$ and a one dimensional summary statistic $m(L_{t-1}) \in \mathbb{R}$ of the log-likelihood in the previous period. I was able to prove convergence under somewhat very restrictive restrictions on $f$. do you know any result that might cover this more specific setup? – Peter Nov 21 '16 at 0:12
• No, because it has the same problem, in the case I outlined $X_1$ is the sufficient statistic, and that is part of the problem because the first observation has all the information you ever get about the parameter. – user83457 Nov 23 '16 at 12:15