# Occupation times for two-state Markov processes

Consider a two-state Markov process in continuous time, with states labelled $$A$$ and $$B$$. The transition rates for going from state $$A$$ to $$B$$, and state $$B$$ to $$A$$ are $$\alpha$$ and $$\beta$$ respectively. We are interested in the quantity $$p_t(A,x,t_A|A)$$, which denotes the probability that, at time $$t$$, the Markov process is in state $$A$$, has made a total of $$x$$ jumps, and has spent time equal to $$t_A$$ in state $$A$$, given that it started from state $$A$$.

I was thinking of two approaches (and both are likely to be equivalent). But I seem to be running into hurdles in both of them, and would be grateful for some help.

My first approach was to write a set of coupled recurrence relations in the following way: \begin{align} p_t(A,x,t_A|A) &= \int_0^t e^{-\alpha t'}\cdot \alpha \cdot p_{t-t'}(A,x-1,t_A-t'|B)\cdot dt'\\ p_t(A,x,t_A|B) &= \int_0^t e^{-\beta t'}\cdot \beta \cdot p_{t-t'}(A,x-1,t_A|A)\cdot dt' \end{align} The basic idea behind the above equations is that the first jump takes place at time $$t'$$, and thus, the rest of the $$x-1$$ jumps have to be made in the remaining $$t-t'$$ amount of time. Similarly, depending on what the initial state was, the time needed to be spent in state $$A$$ in the remaining $$t-t'$$ time is stated in the argument (e.g. if the first $$t'$$ amount of time was spent in state $$A$$, then in the remaining amount of time, only $$t_A-t'$$ amount of time needs to be spent there). There is a minor issue here that the above equation does not necessarily restrict the possibility of $$t'>t_A$$, which is unphysical. But that can be taken care of by changing the upper limit of the integral in the first equation to $$t_A$$.

Naively, one would expect that taking a Laplace transform of the two equations with respect to $$t$$ should simplify matters a lot. For the second equation, it indeed does, as the equation is a simple convolution in $$t$$. However, the first is not (please correct me if I am wrong) and thus, naive Laplace transform with respect to $$t$$ might not be helpful. Is there any other approach which makes the problem easier?

My second approach was to derive the differential equation satisfied by the quantity $$p_t(A,x,t_A|A)$$, by working in discrete time-steps of $$\Delta t$$, and finally taking the limit $$\Delta t \to 0$$:

\begin{align} p_{t+\Delta t}(A,x,t_A|A) = p_t(A,x,t_A-\Delta t|A)\cdot (1-\alpha \Delta t) + p_t(B,x-1,t_A-\Delta t|A)\cdot \beta \Delta t \end{align} which upon rearranging gives us \begin{align} \frac{p_{t+\Delta t}(A,x,t_A|A) - p_{t}(A,x,t_A- \Delta t|A)}{\Delta t} = -\alpha\cdot p_t(A,x,t_A-\Delta t|A) + \beta \cdot p_t(B,x-1,t_A - \Delta t|A). \end{align} In the limit of $$\Delta t \to 0$$, we get: $$\frac{\partial{p_{t}(A,x,t_A|A)}}{\partial{t}}+\frac{\partial{p_{t}(A,x,t_A|A)}}{\partial{t_A}} = -\alpha\cdot p_t(A,x,t_A|A) + \beta \cdot p_t(B,x-1,t_A|A).$$

Similarly, one could write equations for $$p_t(B,x,t_A|A)$$. Here (again naively), it seems like one could take a Laplace transform with respect to $$t$$ and $$t_A$$, and define a generating function for $$x$$ to simplify the problem. However, one thing that scares me is that $$t_A$$ cannot take any value. It must always take values less than $$t$$. Thus, independently taking Laplace transforms with respect to $$t$$ and $$t_A$$ seems shady.

Is there a way to overcome one (or both) of the hurdles?

• Did you try to calculate these functions using your recurrence for $x=1,2,3$? Commented Oct 9, 2021 at 21:37

Without loss of generality, let the final time to be $$t=1$$ (if it is not, we can make it so by rescaling time as $$\alpha'=t\alpha$$ and $$\beta'=t\alpha$$).

Then, consider a single trajectory of the process that starts and ends on state $$A$$ and makes $$x$$ jumps ($$x$$ has to be even). We can represent any such trajectory in terms of the jump times, $$\vec{t}=(t_{0}=0\le t_{1}\le\dots\le t_{x}\le1) \tag{1}$$ The conditional probability of the trajectory is given by the product of the jump probabilities, multiplied by the probability of not leaving state $$A$$ after the last jump: $$p(\vec{t}\vert A)=e^{-\alpha(1-t_{x})}\prod_{i=0}^{x-1}p_{i}(t_{i+1}\vert t_{i}).\tag{2}$$ We will assume below that $$x\ge 2$$ (otherwise $$t_A=1$$ and $$p(A,x,t_A|A)=e^{-\alpha}$$).

Now, if $$i$$ is even, then the jump at $$t_{i+1}$$ is from state $$A$$ to state $$B$$ and has probability density $$p_{i}(t_{i+1}\vert t_{i})=\alpha e^{-(t_{i+1}-t_{i})\alpha}$$. Otherwise the jump is from state $$B$$ to state $$A$$ and has probability density $$p_{i}(t_{i+1}\vert t_{i})=\beta e^{-(t_{i+1}-t_{i})\beta}$$. Plugging into $$(2)$$ and simplifying gives $$p(\vec{t}\vert A)=e^{-[t_{A}(\vec{t})\alpha+(1-t_{A}(\vec{t}))\beta]}\alpha^{x/2}\beta^{x/2}\tag{3},$$ where $$t_A(\vec{t})$$ indicates the amount of time that trajectory $$\vec{t}$$ spends in state $$A$$.

To get your answer, we integrate over the set of all ordered sequences $$\vec{t}$$ that obey $$(1)$$ and spend $$t_A$$ time in state $$A$$, which we indicate as $$\Omega$$. This gives: \begin{align}P(A,x,t_{A}\vert A)=\int_{\Omega}p(\vec{t}\vert A)d\vec{t}=e^{-[t_{A}\alpha+(1-t_{A})\beta]}\alpha^{x/2}\beta^{x/2}\int_{\Omega} d\vec{t}.\tag{4}\end{align}

It remains to calculate the volume of $$\Omega$$. Let us indicate this set as $$\Omega=\{t\in\mathbb{R}_{+}^{x}:0\le t_{1}\le\dots t_{x}\le1,\sum_{i=1}^{x/2}t_{2i}-t_{2i-1}=1-t_{A}\}$$ Note that $$\Omega$$ has the same volume as the following subset of the unit $$x$$-simplex: $$\Omega'=\{g\in\mathbb{R}_{+}^{x+1}:\sum_{i=1}^{x+1} g_{i}=1,\sum_{i=1}^{x/2}g_{2i}=1-t_{A}\},$$ which follows from the simple transformation $$g_{i}=t_{i}-t_{i-1}$$ for $$i=1..x+1$$ (where $$t_0=0$$ and $$t_{x+1}=1$$). Then, by rearranging the odd and even coordinates, it is easy to see that $$\Omega'$$ is itself a cross product of two scaled unit simplices: $$\Omega'=\{g\in\mathbb{R}_{+}^{x/2+1}:\sum_{i=1}^{x/2+1}g_{i}=t_{A}\}\times\{g\in\mathbb{R}_{+}^{x/2}:\sum_{i=1}^{x/2} g_{i}=1-t_{A}\}$$ The volume of $$\Omega'$$ is the product of the volumes of these simplices, $$\mathrm{Vol}(\Omega') =\mathrm{Vol}(\Omega) = \frac{t_{A}^{x/2}}{(x/2)!}\frac{(1-t_{A})^{x/2-1}}{(x/2-1)!}$$ Combined with $$(4)$$ this gives \begin{align}P(A,x,t_{A}\vert A)=e^{-[t_{A}\alpha+(1-t_{A})\beta]}\alpha^{x/2}\beta^{x/2}\frac{t_{A}^{x/2}}{(x/2)!}\frac{(1-t_{A})^{x/2-1}}{(x/2-1)!}.\end{align}

• Thanks a lot for the answer -- I was fairly convinced by your logic. However, I am slightly concerned about the final two steps. When you integrate over all $\Omega$ (all admissible combinations of $t_1, t_2, \dots, t_x$), aren't you integrating over all $t_A$ also? So perhaps $\Omega$ should be restricted in such a way, that $t_A$ remains a constant. I have been trying a different approach to the problem, and will post my solution shortly. Commented Oct 21, 2021 at 9:58
• @StatisticalMechanic I think you are absolutely right, my original answer was not correct because I integrated over $t_A$ -- apologies. However, I think there is a simple fix, which is to represent the set of "constrained" trajectories (where each trajectory is required to spend time $t_A$ in state $A$) is written as a cross product of two simplices. I have updated my answer. Commented Oct 21, 2021 at 17:24
• This is fantastic, thanks! Just one more thing -- we do not want to compute the volume of these simplices, right? Rather, we want their surface area. For example, if we take $x=3$, then we have $g_1,g_2,g_3$ and $g_4$ such that $g_1+g_3=t_A$ and $g_2+g_4=1-t_A$. Now, we are not interested in the area of the right angled triangle formed by the $g_1$ and $g_3$ axes, and the line $g_1+g_3=t_A$. Instead, we want the length of the line with endpoints $(0,t_A)$ and $(t_A,0)$, because that defines the number of ways $g_1$ and $g_3$ can be distributed. Does that make sense? Commented Oct 23, 2021 at 10:35
• Perhaps we will need to use this result: math.stackexchange.com/questions/2996038/… Commented Oct 23, 2021 at 10:39
• @StatisticalMechanic I don't think your counterexample works, as $x$ has to be even for the trajectory to end on state A. Instead, let us take $x=2$ as an example. Then, $g_1+g_3=t_A$ and $g_2=1-t_A$, and the volume of $\Omega'$ is given by the volume of a simplex with two vertices (i.e., a line) and a simplex with one vertex (i.e., a point), as expected. Commented Oct 25, 2021 at 0:42