Markov chain convergence problem. - MathOverflow

Markov chain convergence problem.

2010-09-09T17:32:30Z

Consider a markov chain matrix P of size n x n (n states).

P is known to be:

1- there are at least two absorbent states. one of them is denoted by null. (thus, we have that P_null,null = 1)

2- For the set of states that are not absorbent (called set H) , we have that P_h,null > 0 for all h in H.

3- Not all states are recurrent.

4- Aperiodic (the return to some states can occur at irregular times).

It is true that limit when n goes to infinity of P^n converges? Is this result well known or is the proof simple?

Thanks.

Answer by Bjørn Kjos-Hanssen for Markov chain convergence problem.

2010-09-09T18:04:58Z

Yes. $P^n$ converges to a matrix $Q$ with

(i) $Q_{i,i}=1$ for each $i\not\in H$ ($i$ is absorbent), and

(ii) $\sum_{j\not\in H} Q_{i,j}=1$ for all $i\in H$.

To see (ii) we need that $\sum_{j\not\in H}P^n_{i,j}\rightarrow 1$ for all $i\in H$. For this note that $\sum_{j\not\in H}P^n_{i,j}$ is the probability of going from $i$ to an absorbent state in $n$ steps, and so if $P_{i,\text{null}}\ge\lambda>0$ for all $i\in H$ then for all $j\not\in H$, $$ P^n_{i,j}\le (1-\lambda)^n\rightarrow 0. $$ To get a unique such $Q$ we need to show for each absorbent state (say, for null) $$ \lim_n \ P^n_{i,\text{null}}\quad\text{exists} $$ for each $i\in H$. But $P^n_{i,\text{null}}\le P^{n+1}_{i,\text{null}}$ since once we get to null we stay there.

Answer by Byron Schmuland for Markov chain convergence problem.

2010-09-09T21:42:57Z

Yes, uniqueness holds.

Condition 2 implies that every state $j$ is either absorbing $(j\not\in H)$ or transient $(j\in H)$. Define the absorption time to be $T=\inf (n\geq 0: X_n\not\in H)$. This $T$ is almost surely finite for any starting state $i$, that is, the chain is eventually absorbed.

If $j\in H$, then $p^n_{ij}=P_i(X_n=j)\leq P_i(T>n)\to 0=:Q_{ij}$ as $n\to\infty$.

If $j\not\in H$, then $p^n_{ij}=P_i(X_n=j)\uparrow P_i(X_T=j)=:Q_{ij}$ as $n\to\infty$.