All Questions
141 questions
14
votes
2
answers
2k
views
Markov chains: invariant measures and explosion
The following seems like such an elementary question, but I didn't get anywhere with it.
Suppose you are considering a Markov chain in continuous time which is transient and has an invariant measure (...
12
votes
3
answers
4k
views
How to explain "Feller process" to an undergraduate student?
I had to explain in informal terms what a Feller process was, to undergraduate students who understand Markov property, Poisson processes and such. It was easy to define Levy process as generalisation ...
- 519
11
votes
1
answer
642
views
Random walk origin return monotinicity
Consider a Markov chain on $\mathbb{Z}^d$ with transition kernel $P$ for adjacent vertices (non-diagonal). Essentially this is a $d$ dimensional random walk with the probability of a transition ...
- 4,952
8
votes
4
answers
8k
views
Is there MDPs (Markov Decision Process) which have a non deterministic optimal policy?
I'm working on Markov Decision Process and I have not found yet an example of MDP that has a stochastic (non deterministic) optimal policy. Is there MDPs that have a stochastic optimal policy or is it ...
- 254
8
votes
4
answers
1k
views
Invariant measure of Euler-Maruyama Discretisation of an Ito diffusion
Let $(X_t)_{t \geq 0}$ be a diffusion process with dynamics governed by the stochastic differential equation
\begin{equation}
dX_t = b(X_t)dt + \sigma(X_t)dW_t, ~~ X_0 = x_0,
\end{equation}
where $b,\...
- 203
6
votes
2
answers
2k
views
Random walk to stay in an interval forever
Consider a random walk on the real time, starting from $0$. But this time assume that we can decide, for each step $i$, a step size $t_i>0$ to the left or the right with equal probabilities.
To ...
- 502
6
votes
1
answer
509
views
Approximating Markov chains by Brownian motion
I would like a result along the following lines to be true, but haven't been able to locate it in the literature; pointers would be welcome.
Let $X_{t}$ be a finite-state, irreducible, aperiodic ...
- 1,069
6
votes
1
answer
170
views
Basic Definition and Notations in RWRE
From the definition of Zeitouni's lecture notes on RWRE: $(V, E)$ is a special graph, and $N_v:= \{k \in V: (v,k) \in E\}$ is the neighborhood of $v \in V$. $\Omega = \prod_{v \in V} M_1(N_v)$ ...
- 61
6
votes
2
answers
912
views
References for a physicist migrating to stochastic processes
I've studied "Markov Chains" - Norris and "Measure, Integral and Probability" - Capinski, Kopp. Now, I'm looking for a couple of books (or other references) that help me bridging these two topics. ...
- 161
6
votes
1
answer
387
views
Quasi-stationary distribution for a death process
In the paper, Survival in a quasi-death process by van Doorn and Pollett, the quasi-stationary distribution of a transient CTMC is discussed and QSD for a simple death process is derived.
Consider a ...
- 519
6
votes
1
answer
171
views
Relative vulnerabilities in SIS epidemic model
Consider the SIS model of epidemic spreading. There is a finite graph $G(V,E)$, link infection rates $\lambda_{ij}$ and node recovery rates $\mu_i$. There are a few initial nodes which are infected at ...
- 519
5
votes
2
answers
369
views
Markov process on a torus with prescribed invariant distribution
In Euclidean space, $\mathbb R^d$, the Langevin diffusion $${\rm d}X_t=b(X_t){\rm d}t+\sigma(X_t){\rm d}W_t\tag1,$$ where $\sigma:\mathbb R^d\to\mathbb R^{d\times k}$, $$b:=\frac{\Sigma+U}2\nabla\ln p+...
- 167
5
votes
1
answer
192
views
Non-equivalent definitions of Markov process
As far as I know, there are three definitions of Markov processes (or of Markov chains).
DEFINITION 1 (WEAKER). A process $(X_t)_{t\in[0,\infty)}$ on $(\Omega,\mathcal{F},\mathbb{P})$ with values in ...
- 1,149
5
votes
0
answers
271
views
How to play golf in one dimension?
One-dimensional golf is a function $g$ on $\mathbb R$ such that
$g(x)= 1+\min_\mu E[g(x+N(\mu,c\mu^2))]$ if $|x|>1$ and 0 if $|x|\le 1.$
Here $N$ is the normal distribution, whose mean $\mu$ you ...
- 18.8k
5
votes
0
answers
485
views
Hierarchical Random Walk (also known as Hierarchical Hidden Markov Model)
Let us consider the following hierarchical (recursive) random walk model, which is also known as the hierarchical hidden Markov model in computer science (https://en.wikipedia.org/wiki/...
- 1,127
5
votes
0
answers
95
views
Most visited vertex in a random walk with place dependent drift
Consider the following Markov chain on $\mathbb{Z}$:
$$
P(x,x+1)=1-P(x,x-1)=\frac{1}{2}+e^{-|x|}\cdot \mathbf{1}_{\{x\neq 0\}}
$$
Do there exist constants $c,C>0$ such that
$$
c\cdot P^t(z,z) \...
- 527
4
votes
2
answers
2k
views
Frequency of visiting states in Markov chains
Given a finite, ergodic Markov $\{X_i\}$, and two natural numbers $a>b$. Let
$$p=P\left[\forall n, \sum_{k=n}^{n+a-1} \mathbf{1}_m(X_k)\leq b\right]$$
where $\mathbf{1}_m(X_k) =1$ if $X_k=m$ and 0 ...
- 502
4
votes
1
answer
782
views
A simple problem in markov chains
I'm trying to understand a 1954 paper of Kubo intitled "Note on the stochastic theory of resonance absorption". The specific problem can be stated mathematically as follows: let $X(t)$ be a random ...
4
votes
1
answer
186
views
Population growth with good and evil children - probability good outnumbers evil
Consider the following discrete-time population model. We start with a single "good" individual who reproduces asexually into $k$ children and dies in the process. At generation $t=2$, those ...
- 524
4
votes
1
answer
176
views
Random Walk with "Forward Dependency"
Let $\{X_t\}_{t=-\infty}^{\infty}$ be a sequence of random variables. We are interested in a "random walk" (or more generally, a random field) that can be characterized by
$$
X_t ~|~ X_{t-k}, \ldots, ...
- 1,127
4
votes
2
answers
835
views
Reference on continuous-time finite state filtering
Problem: I'm working in reliability field and have seen papers written on the topic like process of failures when systems are functioning under unobservable (or observable) Markov-like environment, i....
- 267
4
votes
2
answers
255
views
The necessary sufficient condition for recurrence of a Markovian random walk
Suppose $\sigma_{1},\sigma_{2},...$are i.i.d random variables.$S_{0}=0$. Define $S_{n}=S_{0}+\sum_{i=1}^{n}\sigma_{i}$, then ${S_{n}}$ is a Markovian random walk.
I want to figure out the necessary ...
- 41
4
votes
0
answers
264
views
Generalized Markov Processes on CW complexes of dimension > 1
Markov processes have a large variety of applications to physics and chemistry (as well as many other fields). Such processes are formulated on graphs, i.e., CW complexes of dimension one. It is ...
- 18.8k
4
votes
0
answers
282
views
Markov operators and existence of ergodic measures
My question refers to the yesterday's question (see here)
of John Learner and goes as follows:
Can we deduce the existence of an ergodic measure if we know that an invariant measure exists, but the ...
- 141
4
votes
0
answers
1k
views
The spectrum of a Markov Operator and Invariant Measures
Suppose I have a discrete-time Markov Chain (in an infinite dimensional state space $\Omega$) with Markov operator $P$, a linear operator on the space of bounded measurable functions on $\Omega$. (Or ...
- 509
3
votes
4
answers
681
views
Does the variance of a continuous time, time homogeneous, Markov process starting from one point necessarily not decrease?
Let $x_t$ be a zero mean, time homogeneous Markovian process (chiefly look at the case where the value is in $1$ dimension) over time $t$ starting from $x_0=0$. Is it necessary that, in continuous ...
- 2,239
3
votes
2
answers
264
views
Probability of one species reaching zero before the other in a Markov process on a 2d lattice
$\textbf{Background}$: Say we've got a two-variable system of stochastic chemical reactions, with quantities $\vec{x}(t) = (x_1(t),x_2(t)) \in \mathbb{N}^2$ evolving according to the following system, ...
- 41
3
votes
2
answers
436
views
Central limit theorem for weak dependent bernoulli random variables
Suppose $\epsilon_1,\epsilon_2,...$ are i.i.d bounded random variables with compact support. Let $X_k=g_k(\epsilon_k,...,\epsilon_1)$ be Bernoulli random variables with the covariance between $X_i$ ...
- 339
3
votes
2
answers
973
views
How much larger than the relaxation time can the mixing time be?
The notation is mostly taken from the book "Markov chains and mixing times" by Levin, Peres, and Wilmer.
Consider an irreducible, aperiodic, time-reversible, discrete-time Markov chain on a finite ...
- 1,269
3
votes
1
answer
1k
views
Ergodicity of a Markov chain
Hi,
I'd appreciate some help on a Markov chain result I'm trying to show. I believe the following is sufficient for a continuous time Markov chain $(X_t)$ with a countable state space to be ergodic:
...
- 181
3
votes
1
answer
343
views
Positive and Null recurrence of Markov Chains on a General State Space
Suppose $X_n$ is an irreducible, aperiodic and Harris recurrent Markov chain. It is well known that in this case, $X_n$ has a stationary distribution $\pi$.
Are there any conditions that are ...
- 339
3
votes
1
answer
2k
views
Hitting time probability in a Random Walk with possibility to die.
A Random Walker can move of one unit to the right with probability $p$, to the left with probability $q$ and it can jump again to the starting point with probability $r$ and die. Naturally $p+q+r=1$. ...
- 295
3
votes
1
answer
335
views
Stochastic processes having Markov kernels
Let $(\Omega_1, \mathcal{F}_1, P_1)$ and $(\Omega_2, \mathcal{F}_2, P_2)$ be probability spaces and suppose $(X_t)$ and $(Y_t)$ are real-valued stochastic processes defined on the respective spaces. ...
- 33
3
votes
3
answers
1k
views
Markov random field with continuous index set
Hi
There's Markov random field (MRF) which, by my Wikipedia-based knowledge, is an extension of Markov chain. I'd like to think of it as going from 1D to higher dimensional spaces. Inherent in its ...
- 355
3
votes
1
answer
474
views
Harmonic function and Markov chain
Let $X=(X_k)_{k \in \mathbb{N}}$ be a Markov chain with countable countable state space $S$ and transition matrix $P.$
Let $\mathcal{T}$ be the tail $\sigma$-field of $X:\mathcal{T}=\bigcap_{k \in \...
- 53
3
votes
1
answer
182
views
Superlinear Convergence of a Markov Chain
Suppose that we have a Markov process $\{Z_t\}_{t=0}^\infty$, where $Z_t \geq 0$ for any $t$. Assume that, conditioning on $Z_t = z_t$, we have
$
\mathbb{E}\{Z_{t+1}|Z_t = z_t\} \leq \kappa z_t^2
$. ...
- 1,127
3
votes
1
answer
307
views
"Ergodic theorem" for Markov kernels
Consider a discrete time Markov chain $(X_t)$ on a finite state space $\mathcal{S}$, with transition matrix $P$. Assume that the chain admits a stationary distribution $\pi$, which I will identify ...
3
votes
1
answer
340
views
Importance resampling with exponential weighting
Suppose that we have
$$
\frac{p(x)}{q(x)} \propto \exp(\tau f(x)),
$$
where we can sample from $q$ but not from $p$. Our goal is to generate a set of particles $\{x_i\}_{i=1}^n$ such that $n^{-1}\sum_{...
- 1,127
3
votes
1
answer
225
views
Total offspring of Poisson multitype branching process
A normal branching process $Z_n$ initialized with $Z_0=1$ and offspring generated from $Pois(p),p<1,$ has a total progeny / total off spring distribution
$$X=\sum_{n=0}^\infty Z_n$$
$X\in \mathbb{...
- 315
3
votes
2
answers
922
views
On representing a continuous time Markov chain by a stochastic integral of a Poisson random measure
Let $Q=(q_{ij})$ be the transition rate matrix of a continuous time Markov chain $\{ X_t \}$ with countable state space $M$. Let $q_i = -q_{ii}=\sum_{j \neq i}q_{ij}$, and let $\Gamma_{ij}$ be defined ...
- 31
3
votes
1
answer
208
views
Is there an easy way to convert a non-deterministic optimal policy to a deterministic optimal policy for a given MDP?
For a MDP (Markov Decision Process) is there an easy way to convert a non-deterministic optimal policy into a deterministic optimal policy?
The trivial way will take $O(|\mathcal{A}|^{|\mathcal{S}|}$...
- 221
3
votes
3
answers
2k
views
Proof of the existence of an optimal MDP with a stochastic reward signal?
I'm following Sutton's book on Reinforcement Learning, and he casually states that "There is always at least one policy that is better than
or equal to all other policies" for a given finite MDP. This ...
- 133
3
votes
0
answers
241
views
A few questions on Feller processes
Update. Most of my questions have been answered in the comments. I am adding these answers to the post.
There are at least three definitions of Feller semigroup and the corresponding processes: $C_0 \...
- 620
3
votes
0
answers
54
views
Multi-type Galton-Watson-like process where only majority-type is allowed to reproduce
Are you aware of any research papers that have explored a multi-type Galton-Watson process in which only particles of the majority type are permitted to reproduce in each generation?
I've been unable ...
- 56
3
votes
0
answers
90
views
How does one define the gradient of a Markov semigroup?
In the context of functional inequalities for Markov semigroups $(\mathcal P_t)_{t\ge0}$, what is one denoting by $\nabla\mathcal P_tf$? For example, I've found the following assumption in this paper:
...
- 167
3
votes
0
answers
106
views
Find the generator of a markov process with constant decay and exponential jumps
Suppose we have a continuous time Markov process $(X_t)_{t\in [0,\infty)}$. This Markov process represents the queue length in amount of work left, therefore its state space is given as $S = [0,\infty)...
- 277
3
votes
0
answers
305
views
Nonlinear Markov process
Consider the following nonlinear $\mathbb{R}$-valued stochastic recursive sequence:
$ X_{n+1} = F(X_n) + W_{n+1}, \quad (W_n)_{n\ge1} \stackrel{ \scriptsize \mathrm{i.i.d.} }{ \sim } \phi. $
How can ...
- 61
2
votes
1
answer
640
views
Reachability for Markov process
Let $X$ be a Markov process (in continuous or discrete time) and define an event
$$
R(T,A) = (\exists t\leq T: X_t \in A).
$$
I have seen in one paper that
$$
\Pr[R(\infty,A)] = \sup\limits_{\tau} \...
- 1,655
2
votes
1
answer
205
views
Eigenspace of Gaussian Markov operator
Consider the (one-dimensional) Gaussian distribution $Q := N(\nu,\tau^2)$ and the (Gaussian) Markov operator
\begin{equation*}
\begin{array}{rccc}
R : & L_1(\mathbb{R},\mathcal{B}(\mathbb{R}),Q) &...
- 123
2
votes
1
answer
395
views
Probability-one event for Markov chain
Let $X$ be a Markov chain, with countable state space $I$ and transition probability matrix $P$. $X$ is irreducible, but need not be recurrent. Let $S$ be a fixed subset of $I$.
Define a subset $K$ ...
- 1,069