I have a Markov Chain of $N$ states. Such states represent the energy levels in a molecule.

The states' connectivity is as follows:

  1. States $j\in\{0,\ldots,N\}$ transition to $k\in\{\max(j-M,0),...,\min(j+M,N)\}\setminus j$, with $N\gg M$. The resulting transition matrix $\mathbf{P}$ has known terms $P_{jk}$, and it is a hollowed band matrix where detailed balance is followed.

  2. States $l\in\{L,...,N\}$ ALSO transition outside the Markov Chain, with rates $\mu_l>0$. Every state $l$ retains transitions to the $k$ states described in the previous point. For instance, state $L+1$ has both $j$ and $l$ transitions.

I am interested in recovering the stationary distribution $\tilde\pi_j$ of such Markov Chain.

If $\mu_l=0$, the Markov Chain has stationary distribution $\pi_j$ known from detailed balance and probability closure condition: $$ \pi_kP_{kj}=\pi_{j}P_{jk},\quad \sum_j\pi_j=1 $$ writing $\pi_j=\pi_0P_{0j}/P_{j0}$ and substituting into $\sum_j\pi_j=1$, one obtains $\pi_0$ and all the remaining values.

Now, when $\mu_l>0$, the stationary distribution $\tilde\pi_j$ is different. To tackle the problem, I have assumed that the $l$ states go to an absorbing state $J=N+1$, thus the transition matrix gains an absorbing state (one row and one column) with: $$ P_{lJ}=\mu_l,\quad P_{(j\setminus l)J}=0,\quad P_{Jl}=0,\quad P_{J(j\setminus l)}=0,\quad P_{JJ}=1 $$ But from now, to the best of my knowledge, I do not know how to proceed. I have the feeling that by introducing $J$, I am rendering the Markov Chain not ergodic anymore (as all states are unreachable from $J$). I have the feeling that $\tilde \pi_j=\delta(j-J)$, having $\delta(x)$ as the Dirac Delta Function.

I have read various sources (Snell, Montroll, Othmer) on Markov Chains and queuing theory, but all examples were not as specific as the one I have described above.

  • 2
    $\begingroup$ You are right: the unique stationary distribution is concentrated in the absorbing state. This is quite intuitive: random walk on states 1 through $N$ will eventually lead to one of the states $L$ through $N$, and then there is a positive probability of jumping to the absorbing state in one step. Thus, the Markov chain will eventually reach the absorbing state. $\endgroup$ – Mateusz Kwaśnicki Nov 1 '19 at 19:51
  • 1
    $\begingroup$ It can be worth it in cases like this to write a bit of code to simulate a bunch of random walks. $\endgroup$ – usul Nov 2 '19 at 13:56
  • $\begingroup$ it may be of interest that excluding the absorbing states the chain still behaves regularly, the distribution given no absorbtion will converge, and this is a consequence of the perron-frobenius theorem $\endgroup$ – mike Nov 4 '19 at 8:09
  • $\begingroup$ @MateuszKwaśnicki Crystal clear explanation $\endgroup$ – TheVal Nov 16 '19 at 16:55
  • 2
    $\begingroup$ You might be interested in the quasi-stationary distribution of such a Markov chain, see the corresponding Wikipedia page for example. $\endgroup$ – Martin Hairer Nov 16 '19 at 20:44

As the comments state, the resulting Markov Chain must have equilibrium distribution $$ \tilde\pi_j = \delta(j-J) $$ since any random walk through the Chain will eventually lead to states $j\ge L$, and then to $J$. This is justified by the Perron-Frobenius Theorem applied to the transition matrix $\mathbf{P}$.

Alternatively, the inevitable eventuality that a random process will hit $j$ may be given by transitive subshift of finite type applied to the finite oriented graph $G$ generated by $\mathbf{P}$, or the adjacency matrix $\mathbf{A}(\mathbf{P})$ such that: $$ a_{ij}=H(p_{ij}) $$ where $H$ is the Heaviside function $H(x>0)=1$. Since $G$ is strongly connected (there is a sequence of edges from any one vertex to any other vertex), then subshift of finite type on $G$ is transitive.

There may be other tools to describe the existence of an equilibrium distribution for such processes, such as the Krylov-Bogolyubov Theorem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.