All Questions
141 questions
0
votes
1
answer
203
views
Law of large numbers for Harris recurrent Markov chains
I'm trying to familiarize myself with the details of the proof that the Markov chains produce by the Metropolis-Hastings algorithm have a law of large numbers. I've found a half dozen or more ...
- 2,791
2
votes
1
answer
185
views
Entropy rate problem of ergodic Markov process with non-ergodic joint
I have a problem with the entropy rate when two ergodic Markov processes who are independent of each other having a non-ergodic joint. More specifically let us consider two finite-state Markov ...
- 333
1
vote
1
answer
143
views
Comparison of hitting probability of two Markov chains both with only one absorbing state version 2 under stronger condition
Let $N_n:=\{1,2,\cdots,n\}$. Given two finite states Markov chains $\big(X^{(j)}_i\in N_n\}\big)_{i=0}^\infty$ for $j\in\{1,2\}$, both of which have two absorbing states at $1$ and $n$.
$\text{Pr}\...
- 2,239
1
vote
1
answer
170
views
Stationary distribution of Markov Chain with departure
I have a Markov Chain of $N$ states. Such states represent the energy levels in a molecule.
The states' connectivity is as follows:
States $j\in\{0,\ldots,N\}$ transition to $k\in\{\max(j-M,0),...,\...
- 151
0
votes
2
answers
128
views
Markov with epsilon memory and Quantitative Strong Markov property
We have a process $\{X_{t}\}_{t\geq 0}$ ,with fixed parameter $\epsilon>0$, starting from zero that satisfies
The process is strictly monotone $X_{t+r}-X_{t}>0$ with moments existing $p\in(-\...
- 5,474
0
votes
1
answer
80
views
A question about positive operator pregenerator [closed]
Thank you for reading.
My question was raised up when I tried to prove an example in the book of Liggett(1985), which is in P13 Example 2.3(a).
Here is a link of the page:
https://books.google.com/...
- 385
0
votes
2
answers
306
views
Lower bounds on discrete time finite Markov chains hitting probabilities
I am interested in some general theorems related to lower bounds on discrete time finite Markov chains hitting probabilities (preferably ergodic chains , but not necessarily ), with references . ...
3
votes
1
answer
226
views
Total offspring of Poisson multitype branching process
A normal branching process $Z_n$ initialized with $Z_0=1$ and offspring generated from $Pois(p),p<1,$ has a total progeny / total off spring distribution
$$X=\sum_{n=0}^\infty Z_n$$
$X\in \mathbb{...
- 315
2
votes
0
answers
37
views
Exclusion processes from point of view of a tagged particle
I'm interested in the simple exclusion processes on $Z^d$ and the ergodic theorems that can be proved from the point of view of the particle. Ellen Saada proved the following in 1987 (Annals of Prob): ...
- 941
0
votes
1
answer
204
views
How is the Cauchy-Schwarz inequality used in the proof of Lyapunov's criterion in the book "Analysis and Geometry of Markov Diffusion Operators"
Let $(E,\mu,\Gamma)$ be a full Markov triple (see definition below), $J\in\mathcal A$ with $J\ge1$ and $g\in\mathcal A_0$. In the proof of Theorem 4.6.2 of the book "Analysis and Geometry of Markov ...
- 167
1
vote
1
answer
305
views
Existence of a Lyapunov function for a log-concave measure
Let $d\in\mathbb N$, $f:\mathbb R^d\to\mathbb R$ be convex with $$\int e^{-f(x)}\:{\rm d}x<\infty\tag1$$ and $\mu$ denote the measure with density $e^{-f}$ with respect to the Lebesgue measure on $\...
- 167
-1
votes
1
answer
370
views
What's the probability of two independent events in time domain?
Suppose there are two independent events A and B. The probability that A or ...
- 29
2
votes
0
answers
416
views
How can we treat the generator of a discrete-time Markov chain as the generator of a Markov-jump process?
In the popular paper Weak Convergence and Optimal Scaling of Random Walk Metropolis Algorithms by Roberts, Gelman and Gilks, the authors state (see below) that "in the Skorokhod topology, it does not ...
- 167
3
votes
2
answers
264
views
Probability of one species reaching zero before the other in a Markov process on a 2d lattice
$\textbf{Background}$: Say we've got a two-variable system of stochastic chemical reactions, with quantities $\vec{x}(t) = (x_1(t),x_2(t)) \in \mathbb{N}^2$ evolving according to the following system, ...
- 41
1
vote
0
answers
149
views
Construction of Feller's pseudo-poisson process
Let
$(\Omega,\mathcal A,\operatorname P)$ be a probability space
$(E,\mathcal E)$ be a measurable space
$(Y_n)_{n\in\mathbb N_0}$ be a $(E,\mathcal E)$-valued time-homogeneous Markov chain on $(\...
- 167
1
vote
2
answers
302
views
how to derive stationary distribution of maximal entropy random walk
I was reading the paper 0810.4113v2, burda, which analyzed the stationary distribution maximal entropy random walk on the irregular lattice. I am confused on some of the steps.
Description:
The ...
- 211
2
votes
0
answers
37
views
Reference request: semimarkov processes
What are some good modern introductions to the theory of semimarkov processes? To be clear, by a semimarkov processes, I mean a Markov chain, together with "waiting times" between transitions, the ...
- 505
0
votes
0
answers
72
views
Invariant measures for a renewal process driven by Interarrival times bounded away from zero
Good morning, I apologize in advance if my question sounds too basic but after some research I was unable to come up with satisfactory answers to my doubts.
I am currently studying a model which ...
- 277
3
votes
2
answers
436
views
Central limit theorem for weak dependent bernoulli random variables
Suppose $\epsilon_1,\epsilon_2,...$ are i.i.d bounded random variables with compact support. Let $X_k=g_k(\epsilon_k,...,\epsilon_1)$ be Bernoulli random variables with the covariance between $X_i$ ...
- 339
3
votes
1
answer
343
views
Positive and Null recurrence of Markov Chains on a General State Space
Suppose $X_n$ is an irreducible, aperiodic and Harris recurrent Markov chain. It is well known that in this case, $X_n$ has a stationary distribution $\pi$.
Are there any conditions that are ...
- 339
2
votes
1
answer
187
views
Stationary distribution for a Markov Chain on an uncountable space
Suppose $X_n$ are i.i.d. random variables on $\mathbb{R}$ with compact support, and define the Markov chain $Y_n=X_n +\frac{1}{Y_{n-1}}$ on $\Omega=\mathbb{R}\cup \{\infty\}$. Does the chain $Y_n$ ...
- 339
3
votes
0
answers
106
views
Find the generator of a markov process with constant decay and exponential jumps
Suppose we have a continuous time Markov process $(X_t)_{t\in [0,\infty)}$. This Markov process represents the queue length in amount of work left, therefore its state space is given as $S = [0,\infty)...
- 277
3
votes
3
answers
2k
views
Proof of the existence of an optimal MDP with a stochastic reward signal?
I'm following Sutton's book on Reinforcement Learning, and he casually states that "There is always at least one policy that is better than
or equal to all other policies" for a given finite MDP. This ...
- 133
0
votes
0
answers
169
views
Behaviour of a Markov Chain, given a Lyapunov condition
I'm reading this notes from Martin Hairer about convergence of Markov Processes (on a discrete state space $S$ and in continuous time). On page 12, before presenting the so-called "Harris Theorem", ...
- 203
4
votes
1
answer
176
views
Random Walk with "Forward Dependency"
Let $\{X_t\}_{t=-\infty}^{\infty}$ be a sequence of random variables. We are interested in a "random walk" (or more generally, a random field) that can be characterized by
$$
X_t ~|~ X_{t-k}, \ldots, ...
- 1,127
1
vote
1
answer
404
views
Does Irreducibility holds for the Ergodic non-stationary Markov chain?
In the stationary case, I know that if the chain is irreducible and aperiodic, it is Ergodic. But in the non-stationary case, i can not comprehend the content deeply. I want to know if Irreducibility ...
- 121
3
votes
1
answer
182
views
Superlinear Convergence of a Markov Chain
Suppose that we have a Markov process $\{Z_t\}_{t=0}^\infty$, where $Z_t \geq 0$ for any $t$. Assume that, conditioning on $Z_t = z_t$, we have
$
\mathbb{E}\{Z_{t+1}|Z_t = z_t\} \leq \kappa z_t^2
$. ...
- 1,127
2
votes
0
answers
440
views
Hitting time of a specific Markov chain using martingale approach (or otherwise)
Let $0 < c < 1$. Consider the Markov chain $(X_i)$ on $\{0, 1, \dots, n\}$, with transition probabilities
$$ P(k,k+1) = \left(1 - \tfrac {k}{n} \right)(1-c), \quad k = 0, \dots, n-1, $$
$$ P(k,...
- 985
2
votes
0
answers
32
views
$\mbox{Var}(\sum \delta\{X_n > i_n\} )$ i.f.o. correlation of $(X_n)_n$
Question
Suppose we have an ergodic positive stochastic process $(X_n)_{n \in \mathbb{N}}$ (in particular I'm mainly interested in the case where $(X_n)_n$ is an aperiodic, irreducible, positive ...
- 277
0
votes
0
answers
78
views
Core of direct product of Markov processes
Let $X$ and $Y$ be two diffusion processes. Suppose they have generators $G_X$ and $G_Y$ with domains $D(G_X)$ and $D(G_Y)$ and cores $C(G_X)$ and $C(G_Y)$. Let $Z$ be the product diffusion with ...
- 43
3
votes
2
answers
922
views
On representing a continuous time Markov chain by a stochastic integral of a Poisson random measure
Let $Q=(q_{ij})$ be the transition rate matrix of a continuous time Markov chain $\{ X_t \}$ with countable state space $M$. Let $q_i = -q_{ii}=\sum_{j \neq i}q_{ij}$, and let $\Gamma_{ij}$ be defined ...
- 31
1
vote
1
answer
276
views
Number of deaths in birth-death process conditioned on start and end points
Say I have a simple linear continuous time birth-death process with state space the non-negative integers, where there are parameters $b$ and $d$, with the rate (as you'd see in a $Q$ matrix) of going ...
- 11
2
votes
0
answers
74
views
Literature/Book on counting processes
I seek literature that makes a rigorous treatment of counting processes. In particular im interested in a precise treatment of the conditional intensity $\lambda_t$ which is often informally defined ...
- 315
1
vote
1
answer
4k
views
First passage time of a 1D simple random walk in a discrete time infinite markov chain [closed]
If we consider a simple Random Walk on the positive integers (discrete Markov chain), with symmetric transition probabilities. We start at time $0$ at the integer $i_0 = m$ and at each time step $P(...
- 53
2
votes
2
answers
184
views
Asymptotic Growth of Markov Chain
I asked the following question one week ago at math.stackexchange but didn't receive a response, so I want to give it here another try:
I'm interested in the following problem: We have got a time-...
- 111
5
votes
0
answers
485
views
Hierarchical Random Walk (also known as Hierarchical Hidden Markov Model)
Let us consider the following hierarchical (recursive) random walk model, which is also known as the hierarchical hidden Markov model in computer science (https://en.wikipedia.org/wiki/...
- 1,127
2
votes
1
answer
412
views
Does random walk have more concentration surrounding the origin?
Consider a simple random walk $S_n$ on one dimension, starting at $0$. In this case, $S_n$ fluctuates between $-\infty$ and $\infty$, but intuition says that it might stay more often in an interval ...
- 502
4
votes
2
answers
2k
views
Frequency of visiting states in Markov chains
Given a finite, ergodic Markov $\{X_i\}$, and two natural numbers $a>b$. Let
$$p=P\left[\forall n, \sum_{k=n}^{n+a-1} \mathbf{1}_m(X_k)\leq b\right]$$
where $\mathbf{1}_m(X_k) =1$ if $X_k=m$ and 0 ...
- 502
6
votes
2
answers
2k
views
Random walk to stay in an interval forever
Consider a random walk on the real time, starting from $0$. But this time assume that we can decide, for each step $i$, a step size $t_i>0$ to the left or the right with equal probabilities.
To ...
- 502
6
votes
1
answer
170
views
Basic Definition and Notations in RWRE
From the definition of Zeitouni's lecture notes on RWRE: $(V, E)$ is a special graph, and $N_v:= \{k \in V: (v,k) \in E\}$ is the neighborhood of $v \in V$. $\Omega = \prod_{v \in V} M_1(N_v)$ ...
- 61
1
vote
1
answer
222
views
Uniqueness of invariant measure for equivalent transition probabilities
Suppose $P(x,dy)$ and $Q(x,dy)$ are two Markov transition kernels on a topological space $E$ equipped with Borel $\sigma$-algebra $\mathcal B(E)$. Suppose for every $x \in E$, $P(x,\cdot)$ and $Q(x, \...
- 985
2
votes
0
answers
207
views
markov processes and ergodic theory
For an ergodic Markov Chain
$$
\frac{1}{N}\sum_{i=1}^n f(X_i) \rightarrow E_\pi[f]
$$
where $\pi$ is the invariant distribution. I am also dealing with a Markovian process (a state space model to ...
- 169
4
votes
2
answers
255
views
The necessary sufficient condition for recurrence of a Markovian random walk
Suppose $\sigma_{1},\sigma_{2},...$are i.i.d random variables.$S_{0}=0$. Define $S_{n}=S_{0}+\sum_{i=1}^{n}\sigma_{i}$, then ${S_{n}}$ is a Markovian random walk.
I want to figure out the necessary ...
- 41
3
votes
1
answer
208
views
Is there an easy way to convert a non-deterministic optimal policy to a deterministic optimal policy for a given MDP?
For a MDP (Markov Decision Process) is there an easy way to convert a non-deterministic optimal policy into a deterministic optimal policy?
The trivial way will take $O(|\mathcal{A}|^{|\mathcal{S}|}$...
- 221
2
votes
0
answers
166
views
Must rows of a transition matrix be distinct?
Is it true that for all continuous time Markov processes on a countable state space $S$, we have
all rows of the transition matrix $\mathbf{P}_t$ are distinct for all time $t\in[0,\infty)$ ?
This ...
- 165
5
votes
0
answers
95
views
Most visited vertex in a random walk with place dependent drift
Consider the following Markov chain on $\mathbb{Z}$:
$$
P(x,x+1)=1-P(x,x-1)=\frac{1}{2}+e^{-|x|}\cdot \mathbf{1}_{\{x\neq 0\}}
$$
Do there exist constants $c,C>0$ such that
$$
c\cdot P^t(z,z) \...
- 527
2
votes
1
answer
168
views
Random Walk 2D with dependent weights [closed]
I have spent a lot of time trying to solve this problem but have had no luck so far! Any help would be highly appreciated!
Suppose I have a 3x3 grid as shown below.
(3,1) (3,2) (3,3)
(2,1) (2,2) (...
- 23
1
vote
1
answer
108
views
Regularity of the entrance measure of SRW
Let $S(n)$ be the discrete sphere of radius $n$ (i.e., the internal boundary of the Euclidean discrete ball $B(n)$) centered in the origin, and consider a simple random walk starting at some $x\in\...
- 1,897
1
vote
1
answer
370
views
Markov chain with Feller property
Does anybody know whether there is an analysis of when the monotone decreasing chain has the Feller property?
The monotone decreasing is defined as a chain on $\mathbb{N}$ and the rate of going down $...
- 13
0
votes
0
answers
355
views
Summing up costs over a Markov chain
I apologize in advance if this question is too simplistic to be appropriate for MathOverflow. I have inquired in multiple places but have found little to indicate that this is a previously studied ...
- 101