All Questions
141 questions
2
votes
0
answers
175
views
Representing a continuous time-inhomogeneous Markov chain by a stochastic integral
I am interested in the following mean-field model introduced in the reference below:
There are $N$ particles. At each instant of time, a particle's state is a particular value taken from the finite ...
- 31
3
votes
2
answers
264
views
Probability of one species reaching zero before the other in a Markov process on a 2d lattice
$\textbf{Background}$: Say we've got a two-variable system of stochastic chemical reactions, with quantities $\vec{x}(t) = (x_1(t),x_2(t)) \in \mathbb{N}^2$ evolving according to the following system, ...
- 41
3
votes
3
answers
2k
views
Proof of the existence of an optimal MDP with a stochastic reward signal?
I'm following Sutton's book on Reinforcement Learning, and he casually states that "There is always at least one policy that is better than
or equal to all other policies" for a given finite MDP. This ...
- 133
2
votes
0
answers
416
views
How can we treat the generator of a discrete-time Markov chain as the generator of a Markov-jump process?
In the popular paper Weak Convergence and Optimal Scaling of Random Walk Metropolis Algorithms by Roberts, Gelman and Gilks, the authors state (see below) that "in the Skorokhod topology, it does not ...
- 167
1
vote
2
answers
302
views
how to derive stationary distribution of maximal entropy random walk
I was reading the paper 0810.4113v2, burda, which analyzed the stationary distribution maximal entropy random walk on the irregular lattice. I am confused on some of the steps.
Description:
The ...
- 211
8
votes
4
answers
1k
views
Invariant measure of Euler-Maruyama Discretisation of an Ito diffusion
Let $(X_t)_{t \geq 0}$ be a diffusion process with dynamics governed by the stochastic differential equation
\begin{equation}
dX_t = b(X_t)dt + \sigma(X_t)dW_t, ~~ X_0 = x_0,
\end{equation}
where $b,\...
- 203
3
votes
0
answers
90
views
How does one define the gradient of a Markov semigroup?
In the context of functional inequalities for Markov semigroups $(\mathcal P_t)_{t\ge0}$, what is one denoting by $\nabla\mathcal P_tf$? For example, I've found the following assumption in this paper:
...
- 167
2
votes
1
answer
185
views
Entropy rate problem of ergodic Markov process with non-ergodic joint
I have a problem with the entropy rate when two ergodic Markov processes who are independent of each other having a non-ergodic joint. More specifically let us consider two finite-state Markov ...
- 333
0
votes
2
answers
128
views
Markov with epsilon memory and Quantitative Strong Markov property
We have a process $\{X_{t}\}_{t\geq 0}$ ,with fixed parameter $\epsilon>0$, starting from zero that satisfies
The process is strictly monotone $X_{t+r}-X_{t}>0$ with moments existing $p\in(-\...
- 5,474
6
votes
1
answer
509
views
Approximating Markov chains by Brownian motion
I would like a result along the following lines to be true, but haven't been able to locate it in the literature; pointers would be welcome.
Let $X_{t}$ be a finite-state, irreducible, aperiodic ...
- 1,069
1
vote
1
answer
143
views
Comparison of hitting probability of two Markov chains both with only one absorbing state version 2 under stronger condition
Let $N_n:=\{1,2,\cdots,n\}$. Given two finite states Markov chains $\big(X^{(j)}_i\in N_n\}\big)_{i=0}^\infty$ for $j\in\{1,2\}$, both of which have two absorbing states at $1$ and $n$.
$\text{Pr}\...
- 2,239
1
vote
1
answer
305
views
Existence of a Lyapunov function for a log-concave measure
Let $d\in\mathbb N$, $f:\mathbb R^d\to\mathbb R$ be convex with $$\int e^{-f(x)}\:{\rm d}x<\infty\tag1$$ and $\mu$ denote the measure with density $e^{-f}$ with respect to the Lebesgue measure on $\...
- 167
1
vote
1
answer
170
views
Stationary distribution of Markov Chain with departure
I have a Markov Chain of $N$ states. Such states represent the energy levels in a molecule.
The states' connectivity is as follows:
States $j\in\{0,\ldots,N\}$ transition to $k\in\{\max(j-M,0),...,\...
- 151
3
votes
1
answer
226
views
Total offspring of Poisson multitype branching process
A normal branching process $Z_n$ initialized with $Z_0=1$ and offspring generated from $Pois(p),p<1,$ has a total progeny / total off spring distribution
$$X=\sum_{n=0}^\infty Z_n$$
$X\in \mathbb{...
- 315
1
vote
0
answers
61
views
Convergence of empirical measure to Mc-Kean Vlasov equation for mean-field model with jumps
I am interested in the following mean-field model introduced in the reference below:
There are $N$ particles. At each instant of time, a particle's state is a particular value taken from the finite ...
- 31
0
votes
2
answers
306
views
Lower bounds on discrete time finite Markov chains hitting probabilities
I am interested in some general theorems related to lower bounds on discrete time finite Markov chains hitting probabilities (preferably ergodic chains , but not necessarily ), with references . ...
-1
votes
1
answer
370
views
What's the probability of two independent events in time domain?
Suppose there are two independent events A and B. The probability that A or ...
- 29
1
vote
1
answer
4k
views
First passage time of a 1D simple random walk in a discrete time infinite markov chain [closed]
If we consider a simple Random Walk on the positive integers (discrete Markov chain), with symmetric transition probabilities. We start at time $0$ at the integer $i_0 = m$ and at each time step $P(...
- 53
3
votes
2
answers
436
views
Central limit theorem for weak dependent bernoulli random variables
Suppose $\epsilon_1,\epsilon_2,...$ are i.i.d bounded random variables with compact support. Let $X_k=g_k(\epsilon_k,...,\epsilon_1)$ be Bernoulli random variables with the covariance between $X_i$ ...
- 339
0
votes
1
answer
204
views
How is the Cauchy-Schwarz inequality used in the proof of Lyapunov's criterion in the book "Analysis and Geometry of Markov Diffusion Operators"
Let $(E,\mu,\Gamma)$ be a full Markov triple (see definition below), $J\in\mathcal A$ with $J\ge1$ and $g\in\mathcal A_0$. In the proof of Theorem 4.6.2 of the book "Analysis and Geometry of Markov ...
- 167
3
votes
2
answers
973
views
How much larger than the relaxation time can the mixing time be?
The notation is mostly taken from the book "Markov chains and mixing times" by Levin, Peres, and Wilmer.
Consider an irreducible, aperiodic, time-reversible, discrete-time Markov chain on a finite ...
- 1,269
12
votes
3
answers
4k
views
How to explain "Feller process" to an undergraduate student?
I had to explain in informal terms what a Feller process was, to undergraduate students who understand Markov property, Poisson processes and such. It was easy to define Levy process as generalisation ...
- 519
4
votes
2
answers
2k
views
Frequency of visiting states in Markov chains
Given a finite, ergodic Markov $\{X_i\}$, and two natural numbers $a>b$. Let
$$p=P\left[\forall n, \sum_{k=n}^{n+a-1} \mathbf{1}_m(X_k)\leq b\right]$$
where $\mathbf{1}_m(X_k) =1$ if $X_k=m$ and 0 ...
- 502
3
votes
1
answer
343
views
Positive and Null recurrence of Markov Chains on a General State Space
Suppose $X_n$ is an irreducible, aperiodic and Harris recurrent Markov chain. It is well known that in this case, $X_n$ has a stationary distribution $\pi$.
Are there any conditions that are ...
- 339
0
votes
1
answer
80
views
A question about positive operator pregenerator [closed]
Thank you for reading.
My question was raised up when I tried to prove an example in the book of Liggett(1985), which is in P13 Example 2.3(a).
Here is a link of the page:
https://books.google.com/...
- 385
4
votes
2
answers
835
views
Reference on continuous-time finite state filtering
Problem: I'm working in reliability field and have seen papers written on the topic like process of failures when systems are functioning under unobservable (or observable) Markov-like environment, i....
- 267
4
votes
1
answer
176
views
Random Walk with "Forward Dependency"
Let $\{X_t\}_{t=-\infty}^{\infty}$ be a sequence of random variables. We are interested in a "random walk" (or more generally, a random field) that can be characterized by
$$
X_t ~|~ X_{t-k}, \ldots, ...
- 1,127
1
vote
1
answer
370
views
Markov chain with Feller property
Does anybody know whether there is an analysis of when the monotone decreasing chain has the Feller property?
The monotone decreasing is defined as a chain on $\mathbb{N}$ and the rate of going down $...
- 13
0
votes
0
answers
355
views
Summing up costs over a Markov chain
I apologize in advance if this question is too simplistic to be appropriate for MathOverflow. I have inquired in multiple places but have found little to indicate that this is a previously studied ...
- 101
1
vote
1
answer
404
views
Does Irreducibility holds for the Ergodic non-stationary Markov chain?
In the stationary case, I know that if the chain is irreducible and aperiodic, it is Ergodic. But in the non-stationary case, i can not comprehend the content deeply. I want to know if Irreducibility ...
- 121
1
vote
0
answers
149
views
Construction of Feller's pseudo-poisson process
Let
$(\Omega,\mathcal A,\operatorname P)$ be a probability space
$(E,\mathcal E)$ be a measurable space
$(Y_n)_{n\in\mathbb N_0}$ be a $(E,\mathcal E)$-valued time-homogeneous Markov chain on $(\...
- 167
2
votes
1
answer
187
views
Stationary distribution for a Markov Chain on an uncountable space
Suppose $X_n$ are i.i.d. random variables on $\mathbb{R}$ with compact support, and define the Markov chain $Y_n=X_n +\frac{1}{Y_{n-1}}$ on $\Omega=\mathbb{R}\cup \{\infty\}$. Does the chain $Y_n$ ...
- 339
3
votes
1
answer
182
views
Superlinear Convergence of a Markov Chain
Suppose that we have a Markov process $\{Z_t\}_{t=0}^\infty$, where $Z_t \geq 0$ for any $t$. Assume that, conditioning on $Z_t = z_t$, we have
$
\mathbb{E}\{Z_{t+1}|Z_t = z_t\} \leq \kappa z_t^2
$. ...
- 1,127
5
votes
0
answers
485
views
Hierarchical Random Walk (also known as Hierarchical Hidden Markov Model)
Let us consider the following hierarchical (recursive) random walk model, which is also known as the hierarchical hidden Markov model in computer science (https://en.wikipedia.org/wiki/...
- 1,127
2
votes
0
answers
440
views
Hitting time of a specific Markov chain using martingale approach (or otherwise)
Let $0 < c < 1$. Consider the Markov chain $(X_i)$ on $\{0, 1, \dots, n\}$, with transition probabilities
$$ P(k,k+1) = \left(1 - \tfrac {k}{n} \right)(1-c), \quad k = 0, \dots, n-1, $$
$$ P(k,...
- 985
2
votes
1
answer
412
views
Does random walk have more concentration surrounding the origin?
Consider a simple random walk $S_n$ on one dimension, starting at $0$. In this case, $S_n$ fluctuates between $-\infty$ and $\infty$, but intuition says that it might stay more often in an interval ...
- 502
0
votes
0
answers
169
views
Behaviour of a Markov Chain, given a Lyapunov condition
I'm reading this notes from Martin Hairer about convergence of Markov Processes (on a discrete state space $S$ and in continuous time). On page 12, before presenting the so-called "Harris Theorem", ...
- 203
6
votes
1
answer
171
views
Relative vulnerabilities in SIS epidemic model
Consider the SIS model of epidemic spreading. There is a finite graph $G(V,E)$, link infection rates $\lambda_{ij}$ and node recovery rates $\mu_i$. There are a few initial nodes which are infected at ...
- 519
2
votes
0
answers
37
views
Exclusion processes from point of view of a tagged particle
I'm interested in the simple exclusion processes on $Z^d$ and the ergodic theorems that can be proved from the point of view of the particle. Ellen Saada proved the following in 1987 (Annals of Prob): ...
- 941
3
votes
4
answers
681
views
Does the variance of a continuous time, time homogeneous, Markov process starting from one point necessarily not decrease?
Let $x_t$ be a zero mean, time homogeneous Markovian process (chiefly look at the case where the value is in $1$ dimension) over time $t$ starting from $x_0=0$. Is it necessary that, in continuous ...
- 2,239
3
votes
0
answers
106
views
Find the generator of a markov process with constant decay and exponential jumps
Suppose we have a continuous time Markov process $(X_t)_{t\in [0,\infty)}$. This Markov process represents the queue length in amount of work left, therefore its state space is given as $S = [0,\infty)...
- 277
0
votes
0
answers
72
views
Invariant measures for a renewal process driven by Interarrival times bounded away from zero
Good morning, I apologize in advance if my question sounds too basic but after some research I was unable to come up with satisfactory answers to my doubts.
I am currently studying a model which ...
- 277
14
votes
2
answers
2k
views
Markov chains: invariant measures and explosion
The following seems like such an elementary question, but I didn't get anywhere with it.
Suppose you are considering a Markov chain in continuous time which is transient and has an invariant measure (...
1
vote
1
answer
276
views
Number of deaths in birth-death process conditioned on start and end points
Say I have a simple linear continuous time birth-death process with state space the non-negative integers, where there are parameters $b$ and $d$, with the rate (as you'd see in a $Q$ matrix) of going ...
- 11
6
votes
1
answer
170
views
Basic Definition and Notations in RWRE
From the definition of Zeitouni's lecture notes on RWRE: $(V, E)$ is a special graph, and $N_v:= \{k \in V: (v,k) \in E\}$ is the neighborhood of $v \in V$. $\Omega = \prod_{v \in V} M_1(N_v)$ ...
- 61
1
vote
1
answer
222
views
Uniqueness of invariant measure for equivalent transition probabilities
Suppose $P(x,dy)$ and $Q(x,dy)$ are two Markov transition kernels on a topological space $E$ equipped with Borel $\sigma$-algebra $\mathcal B(E)$. Suppose for every $x \in E$, $P(x,\cdot)$ and $Q(x, \...
- 985
4
votes
2
answers
255
views
The necessary sufficient condition for recurrence of a Markovian random walk
Suppose $\sigma_{1},\sigma_{2},...$are i.i.d random variables.$S_{0}=0$. Define $S_{n}=S_{0}+\sum_{i=1}^{n}\sigma_{i}$, then ${S_{n}}$ is a Markovian random walk.
I want to figure out the necessary ...
- 41
2
votes
2
answers
184
views
Asymptotic Growth of Markov Chain
I asked the following question one week ago at math.stackexchange but didn't receive a response, so I want to give it here another try:
I'm interested in the following problem: We have got a time-...
- 111
3
votes
1
answer
208
views
Is there an easy way to convert a non-deterministic optimal policy to a deterministic optimal policy for a given MDP?
For a MDP (Markov Decision Process) is there an easy way to convert a non-deterministic optimal policy into a deterministic optimal policy?
The trivial way will take $O(|\mathcal{A}|^{|\mathcal{S}|}$...
- 221
3
votes
1
answer
2k
views
Hitting time probability in a Random Walk with possibility to die.
A Random Walker can move of one unit to the right with probability $p$, to the left with probability $q$ and it can jump again to the starting point with probability $r$ and die. Naturally $p+q+r=1$. ...
- 295