# Stationary distribution of Markov Chain with departure

I have a Markov Chain of $$N$$ states. Such states represent the energy levels in a molecule.

The states' connectivity is as follows:

1. States $$j\in\{0,\ldots,N\}$$ transition to $$k\in\{\max(j-M,0),...,\min(j+M,N)\}\setminus j$$, with $$N\gg M$$. The resulting transition matrix $$\mathbf{P}$$ has known terms $$P_{jk}$$, and it is a hollowed band matrix where detailed balance is followed.

2. States $$l\in\{L,...,N\}$$ ALSO transition outside the Markov Chain, with rates $$\mu_l>0$$. Every state $$l$$ retains transitions to the $$k$$ states described in the previous point. For instance, state $$L+1$$ has both $$j$$ and $$l$$ transitions.

I am interested in recovering the stationary distribution $$\tilde\pi_j$$ of such Markov Chain.

If $$\mu_l=0$$, the Markov Chain has stationary distribution $$\pi_j$$ known from detailed balance and probability closure condition: $$\pi_kP_{kj}=\pi_{j}P_{jk},\quad \sum_j\pi_j=1$$ writing $$\pi_j=\pi_0P_{0j}/P_{j0}$$ and substituting into $$\sum_j\pi_j=1$$, one obtains $$\pi_0$$ and all the remaining values.

Now, when $$\mu_l>0$$, the stationary distribution $$\tilde\pi_j$$ is different. To tackle the problem, I have assumed that the $$l$$ states go to an absorbing state $$J=N+1$$, thus the transition matrix gains an absorbing state (one row and one column) with: $$P_{lJ}=\mu_l,\quad P_{(j\setminus l)J}=0,\quad P_{Jl}=0,\quad P_{J(j\setminus l)}=0,\quad P_{JJ}=1$$ But from now, to the best of my knowledge, I do not know how to proceed. I have the feeling that by introducing $$J$$, I am rendering the Markov Chain not ergodic anymore (as all states are unreachable from $$J$$). I have the feeling that $$\tilde \pi_j=\delta(j-J)$$, having $$\delta(x)$$ as the Dirac Delta Function.

I have read various sources (Snell, Montroll, Othmer) on Markov Chains and queuing theory, but all examples were not as specific as the one I have described above.

• You are right: the unique stationary distribution is concentrated in the absorbing state. This is quite intuitive: random walk on states 1 through $N$ will eventually lead to one of the states $L$ through $N$, and then there is a positive probability of jumping to the absorbing state in one step. Thus, the Markov chain will eventually reach the absorbing state. – Mateusz Kwaśnicki Nov 1 '19 at 19:51
• It can be worth it in cases like this to write a bit of code to simulate a bunch of random walks. – usul Nov 2 '19 at 13:56
• it may be of interest that excluding the absorbing states the chain still behaves regularly, the distribution given no absorbtion will converge, and this is a consequence of the perron-frobenius theorem – mike Nov 4 '19 at 8:09
• @MateuszKwaśnicki Crystal clear explanation – TheVal Nov 16 '19 at 16:55
• You might be interested in the quasi-stationary distribution of such a Markov chain, see the corresponding Wikipedia page for example. – Martin Hairer Nov 16 '19 at 20:44

As the comments state, the resulting Markov Chain must have equilibrium distribution $$\tilde\pi_j = \delta(j-J)$$ since any random walk through the Chain will eventually lead to states $$j\ge L$$, and then to $$J$$. This is justified by the Perron-Frobenius Theorem applied to the transition matrix $$\mathbf{P}$$.

Alternatively, the inevitable eventuality that a random process will hit $$j$$ may be given by transitive subshift of finite type applied to the finite oriented graph $$G$$ generated by $$\mathbf{P}$$, or the adjacency matrix $$\mathbf{A}(\mathbf{P})$$ such that: $$a_{ij}=H(p_{ij})$$ where $$H$$ is the Heaviside function $$H(x>0)=1$$. Since $$G$$ is strongly connected (there is a sequence of edges from any one vertex to any other vertex), then subshift of finite type on $$G$$ is transitive.

There may be other tools to describe the existence of an equilibrium distribution for such processes, such as the Krylov-Bogolyubov Theorem.