# Does every (generalized?) Markov chain admit transition probabilities?

To pose the question let us start by recalling the following notions:

• Transition Probabilities. A transition probability matrix between two measurable spaces $(S,\mathcal{S})$ and $(V,\mathcal{V})$ is a function $P:S\times\mathcal{V}\to [0,1]$ such that for every $s\in S$, $P(s,\cdot)$ is a probability measure in $\mathcal{V}$ and for every $A\in\mathcal{V}$, $P(\cdot,A)$ is $\mathcal{S}-$measurable. If $(S,\mathcal{S})=(V,\mathcal{V})$ we will speak of a transition probability matrix in $(S,\mathcal{S})$.

• Random Elements. A random element of a measurable space $(S,\mathcal{S})$ is a $\mathcal{F}/\mathcal{S}$ measurable function $\xi:\Omega\to S$ defined on some probability space $(\Omega,\mathcal{F},\mathbb{P})$ (we can identify two random elements if they coincide at $\mathbb{P}-$a.e. $\omega$, but this definition will be enough for our purposes). Note that, in this definition, the word random comes from the introduction of a probability measure in $(\Omega,\mathcal{F})$, but this measure has no role in the property defining $\xi$.

• Markov Chains (with fixed state space). A sequence $(\xi_{k})_{k\in\mathbb{Z}}$ of random elements of $(S,\mathcal{S})$ defined on the same probability space $(\Omega,\mathcal{F},\mathbb{P})$ is a Markov chain if for every $k\in\mathbb{Z}$ there exists a transition probability matrix $P_{k}$ in $(S,\mathcal{S})$ such that for every $A\in\mathcal{S}$, $$\omega\mapsto {P}_{k}(\xi_{k}(\omega),A)$$ defines a version of $\mathbb{P}[\xi_{k+1}\in A|\sigma(\xi_{j})_{j\leq k}]:= E[I_{A}\circ \xi_{k+1}| \sigma(\xi_{j})_{j\leq k}]$ (the conditional expectation is taken with respect to $\mathbb{P}$), where $I_{A}$ is the indicator (or "characteristic'') function of $A$. This implies in particular that for every $A\in\mathcal{S}$ and every $k\in\mathbb{Z}$ $$\mathbb{P}[\xi_{k+1}\in A|\sigma(\xi_{j})_{j\leq k}]=\mathbb{P}[\xi_{k+1}\in A|\sigma(\xi_{k})].$$

• Generalized (?) Markov Chains. If we can verify the last equation (regardless of whether the family of transitions matrices $(P_{k})_{k\in\mathbb{Z}}$ as before exists), we will say that $(\xi_{k})_{k\in\mathbb{Z}}$ is a generalized Markov chain.

Question: Is every generalized Markov chain a Markov chain? I.e. can we always find, for a generalized Markov chain, a family of transition probability matrices $(P_{k})_{k\in\mathbb{Z}}$ such that $\omega\mapsto P_{k}(\xi_{k}(\omega),A)$ is a version of $\mathbb{P}[\xi_{k+1}\in A|\sigma(\xi_{j})_{j\leq k}]$?

Again, I don't know if this question is elementary or its answer is well-known. If such is the case, I would appreciate a reference/explanation of the answer before it gets closed. Note also that the question can be generalized easily to the case in which the state spaces vary with $k$, but an answer to this version is sufficient for my purposes.

• I think you're looking for the existence of a regular conditional probability. You need some mild conditions on the state space $(S, \mathcal{S})$ to ensure its existence. For instance, it suffices to have $(S, \mathcal{S})$ be standard Borel. – Nate Eldredge Mar 21 '16 at 1:01
• Presumably you would want to have some good theorems for these general cases. When do stationary distributions exist? What is the generalized notion of "irreducible"? If a stationary distribution exists, does the system converge to this distribution in any sense, and does it depend on initial conditions? I think some of these are addressed in a book by Meyn and Tweedie. probability.ca/MT/BOOK.pdf – Michael Mar 21 '16 at 20:06
• Of course, the analysis is more complex in comparison to the discrete state space case. Another approach is to avoid Markov analysis by using drift-based theorems and (super) martingale theorems that do not require Markov chains. – Michael Mar 21 '16 at 20:16

Under mild conditions on the state space $(S, \mathcal{S})$, it is true. For instance, it is sufficient that it be standard Borel. In that case, each $\xi_k$ admits a regular conditional probability, which is (in your language) a transition probability matrix $Q_k$ such that for each event $B \in \mathcal{F}$, we have $\mathcal{P}(B \mid \xi_k) = Q_k(\xi_k, B)$ almost surely. Then we can simply let $P_k(x, A) = Q_k(x, \xi_{k+1}^{-1}(A))$.

The existence of regular conditional probabilities is discussed in many advanced probability texts. For instance, you can find it in Durrett's Probability: Theory and Examples, 4e in Section 5.1.3 (for standard Borel spaces). If you want a more general version, the standard place to look is Dellacherie and Meyer, or Bogachev's Measure Theory section 10.4. For instance, I believe regular conditional probabilities still exist if $(S, \mathcal{S})$ is merely Radon; i.e. $\mathcal{S}$ is the Borel $\sigma$-algebra of some topology on $S$ for which every Borel probability measure is Radon.

The result is also a special case of the disintegration theorem, which you can also find treated in depth in the aforementioned references.

The case where the state spaces $S_k$ are distinct also follows, simply by letting $S$ be the disjoint union of all the $S_k$.

But I don't think it's true in full generality. I think the counterexample from Stoyanov's Counterexamples in Probability can be adapted into a generalized Markov chain. If I ever have some spare time I may try to work it out, or anyone else is welcome to do so.

• I have some familiarity with the notion of regular conditional probability. There is a subtle difference though: (here) the transition matrices are defined on the state space. As a matter of fact my question came in part from the fact that people write things like $P(X_{1}|X_{0}=x)$ all the time when talking about Markov chains, but such things are not clearly defined sometimes. I imagine that, after all, both notions are basically the same after one changes the underlying probability space by the state space with the law induced by the elements of the chain. – David Barrera Mar 21 '16 at 15:59
• @DavidBarrera: Right, but as I mentioned, it's simple to move from the sample space to the state space, by pushing forward under $\xi_{k+1}$. Actually, the notion I probably should have discussed instead is that of a regular conditional distribution, which can live on the state space instead. – Nate Eldredge Mar 21 '16 at 18:50
• Yes, though I have the feeling that regular conditional distributions are not always defined as transition probability matrices (or "Markov Kernels", if you prefer). This is, if you want, the "technical meat" behind the (hopefully correct) argument given in the answer below: one needs to ensure that the transition measures are indeed probability measures. The question is scattered around (see for instance this link). – David Barrera Mar 21 '16 at 19:16
• I wonder now if all measurable spaces with countably generated sigma-algebras are actually isomorphic to standard Borel spaces, since this seems to be the restriction present along the results proving the existence of transition probability matrices. – David Barrera Mar 21 '16 at 19:17
• @DavidBarrera: Almost. The result is that if $(S, \mathcal{S})$ is countably generated and also separates points (i.e. for any two points $x,y \in S$, there is a measurable set containing one but not the other; this is obviously necessary), then it is isomorphic to a subspace of a standard Borel space. Equivalently, $\mathcal{S}$ is the Borel $\sigma$-algebra of a separable metric on $S$ (not necessarily complete). This theorem appears in most descriptive set theory texts; see for instance Chapter 12 of Kechris. – Nate Eldredge Mar 21 '16 at 19:54