All Questions
Tagged with pr.probability markov-chains
362 questions
2
votes
2
answers
184
views
Asymptotic Growth of Markov Chain
I asked the following question one week ago at math.stackexchange but didn't receive a response, so I want to give it here another try:
I'm interested in the following problem: We have got a time-...
- 111
5
votes
0
answers
485
views
Hierarchical Random Walk (also known as Hierarchical Hidden Markov Model)
Let us consider the following hierarchical (recursive) random walk model, which is also known as the hierarchical hidden Markov model in computer science (https://en.wikipedia.org/wiki/...
- 1,127
12
votes
3
answers
1k
views
How to sample a uniform random polyomino?
A polyomino is formed by joining finitely many unit squares edge to edge. It may be regarded as a finite subset of the regular square tiling with a connected interior. In particular, for us, ...
- 7,857
2
votes
1
answer
412
views
Does random walk have more concentration surrounding the origin?
Consider a simple random walk $S_n$ on one dimension, starting at $0$. In this case, $S_n$ fluctuates between $-\infty$ and $\infty$, but intuition says that it might stay more often in an interval ...
- 502
4
votes
2
answers
2k
views
Frequency of visiting states in Markov chains
Given a finite, ergodic Markov $\{X_i\}$, and two natural numbers $a>b$. Let
$$p=P\left[\forall n, \sum_{k=n}^{n+a-1} \mathbf{1}_m(X_k)\leq b\right]$$
where $\mathbf{1}_m(X_k) =1$ if $X_k=m$ and 0 ...
- 502
6
votes
2
answers
2k
views
Random walk to stay in an interval forever
Consider a random walk on the real time, starting from $0$. But this time assume that we can decide, for each step $i$, a step size $t_i>0$ to the left or the right with equal probabilities.
To ...
- 502
6
votes
1
answer
170
views
Basic Definition and Notations in RWRE
From the definition of Zeitouni's lecture notes on RWRE: $(V, E)$ is a special graph, and $N_v:= \{k \in V: (v,k) \in E\}$ is the neighborhood of $v \in V$. $\Omega = \prod_{v \in V} M_1(N_v)$ ...
- 61
0
votes
1
answer
414
views
Exact formula for computing n-step transition probability of random walks with self-transitions
Consider a semi-infinite random walks $X_n$, $n=0,1,2,\ldots$, whose state space is a set of consecutive integers and whose one-step transition probabilities are $P_{ij}=\mathrm{Pr}\{X_{n+1}=j|X_n=i\}$...
- 1
1
vote
1
answer
222
views
Uniqueness of invariant measure for equivalent transition probabilities
Suppose $P(x,dy)$ and $Q(x,dy)$ are two Markov transition kernels on a topological space $E$ equipped with Borel $\sigma$-algebra $\mathcal B(E)$. Suppose for every $x \in E$, $P(x,\cdot)$ and $Q(x, \...
- 985
4
votes
3
answers
792
views
Why does the overhand shuffle converge to the uniform distribution on $S_n$?
Pemantle 1989 proves, among other things, that the Markov chain on $S_n$ induced by repeatedly and independently performing an overhand shuffle on a deck of $n$ cards is ergodic and has limiting ...
- 623
2
votes
0
answers
207
views
markov processes and ergodic theory
For an ergodic Markov Chain
$$
\frac{1}{N}\sum_{i=1}^n f(X_i) \rightarrow E_\pi[f]
$$
where $\pi$ is the invariant distribution. I am also dealing with a Markovian process (a state space model to ...
- 169
4
votes
2
answers
255
views
The necessary sufficient condition for recurrence of a Markovian random walk
Suppose $\sigma_{1},\sigma_{2},...$are i.i.d random variables.$S_{0}=0$. Define $S_{n}=S_{0}+\sum_{i=1}^{n}\sigma_{i}$, then ${S_{n}}$ is a Markovian random walk.
I want to figure out the necessary ...
- 41
7
votes
1
answer
621
views
Does every (generalized?) Markov chain admit transition probabilities?
To pose the question let us start by recalling the following notions:
Transition Probabilities. A transition probability matrix between two measurable spaces $(S,\mathcal{S})$ and $(V,\mathcal{V})$ ...
- 486
3
votes
1
answer
208
views
Is there an easy way to convert a non-deterministic optimal policy to a deterministic optimal policy for a given MDP?
For a MDP (Markov Decision Process) is there an easy way to convert a non-deterministic optimal policy into a deterministic optimal policy?
The trivial way will take $O(|\mathcal{A}|^{|\mathcal{S}|}$...
- 221
2
votes
0
answers
166
views
Must rows of a transition matrix be distinct?
Is it true that for all continuous time Markov processes on a countable state space $S$, we have
all rows of the transition matrix $\mathbf{P}_t$ are distinct for all time $t\in[0,\infty)$ ?
This ...
- 165
5
votes
0
answers
95
views
Most visited vertex in a random walk with place dependent drift
Consider the following Markov chain on $\mathbb{Z}$:
$$
P(x,x+1)=1-P(x,x-1)=\frac{1}{2}+e^{-|x|}\cdot \mathbf{1}_{\{x\neq 0\}}
$$
Do there exist constants $c,C>0$ such that
$$
c\cdot P^t(z,z) \...
- 527
1
vote
0
answers
56
views
Expected number of forward jumps to reach a given quantile of a rv [closed]
I'm a noob in randomized algorithm and ran into a problem(definitely not home work. I'm doing a self study out of my interest with help of my friends. I'm pursuing research career in a machine ...
- 221
2
votes
1
answer
168
views
Random Walk 2D with dependent weights [closed]
I have spent a lot of time trying to solve this problem but have had no luck so far! Any help would be highly appreciated!
Suppose I have a 3x3 grid as shown below.
(3,1) (3,2) (3,3)
(2,1) (2,2) (...
- 23
0
votes
0
answers
124
views
Maximal inequality for Markov process
For a Markov process $\{X_n\}$ is there any inequality available for
$$ E[\sup_{0 \leq n \leq k} X_{n}]$$
in terms of moments of $E[X_n], 0 \leq n \leq k$
- 317
1
vote
0
answers
363
views
Markov Chains and Simple Machine Learning [closed]
Suppose I have a large training set consisting of many strings of symbols.
$TS = \{Str_0, Str_1, ..., Str_n\}$
$Str_i = \{Sym_0 ... Sym_{len}\}$
These strings of symbols are each generated by the ...
- 111
1
vote
0
answers
30
views
Properties of a map regarding the space of invariant probability measures for controlled Markov process
Let us consider a controlled Markov process with the transition kernel $p(dy|x,\theta)$ ($\theta$ being the control parameter. Now, consider the map
$\theta \to I(\theta)$ where $I(\theta)$ is the ...
- 317
1
vote
1
answer
108
views
Regularity of the entrance measure of SRW
Let $S(n)$ be the discrete sphere of radius $n$ (i.e., the internal boundary of the Euclidean discrete ball $B(n)$) centered in the origin, and consider a simple random walk starting at some $x\in\...
- 1,897
1
vote
1
answer
370
views
Markov chain with Feller property
Does anybody know whether there is an analysis of when the monotone decreasing chain has the Feller property?
The monotone decreasing is defined as a chain on $\mathbb{N}$ and the rate of going down $...
- 13
8
votes
1
answer
174
views
Equalizing Geometric means of Graph Cycles
Consider a strongly connected directed graph $G$. I have been stuck on the following question: can you assign real numbers in $[0,1]$ to each edge of $G$ so that the geometric mean of all cycles are ...
- 183
3
votes
1
answer
531
views
positive Harris recurrent, aperiodic, stationary Markov chain
How to proof that every positive Harris recurrent, aperiodic, stationary Markov chain is alpha-mixing (strong-mixing)?
- 31
1
vote
1
answer
242
views
Two types of random walkers on square lattice
Consider a two dimensional square lattice ($n$ by $n$), which is our space $S$ (each point labelled by an index $1\to n^2$), containing two types of particles, distinguished here by either an index $1$...
user82424
0
votes
0
answers
355
views
Summing up costs over a Markov chain
I apologize in advance if this question is too simplistic to be appropriate for MathOverflow. I have inquired in multiple places but have found little to indicate that this is a previously studied ...
- 101
3
votes
2
answers
973
views
How much larger than the relaxation time can the mixing time be?
The notation is mostly taken from the book "Markov chains and mixing times" by Levin, Peres, and Wilmer.
Consider an irreducible, aperiodic, time-reversible, discrete-time Markov chain on a finite ...
- 1,269
1
vote
0
answers
44
views
Validating a probability density distribution forecast model for a Markov process
Let's say we have a Markov process $X_t$, and we come up with a forecast model that takes some information from outside world and says: "value $X_{t+1}$ has probability density distribution $P_t(x)$". ...
- 93
3
votes
0
answers
144
views
The spring Markov chain on $\mathbb{N}$
I'm trying to understand and learn more about "almost surely bounded" Markov chains on countable state spaces. I'm looking for references where I can learn how to work with more complicated examples ...
- 1,961
0
votes
0
answers
117
views
Ergodicity property for continuous-time Harris positive Markov process
I have posted this question on there, but got no answer.
The following theorem is Theorem 13.3.3 of Meyn and Tweedie's Markov Chains and Stochastic Stability on page 328:
Theorem 13.3.3. If $\Phi$ ...
- 109
2
votes
1
answer
2k
views
Stationary distribution for time-inhomogeneous Markov process
I have a two state, discrete time, time-inhomogeneous Markov process with transition matrix defined by
$$T_i=\begin{pmatrix}
1-p_i\alpha & p_i\alpha \\
p_i\beta& 1-p_i\beta
\end{pmatrix}$$
...
- 23
1
vote
1
answer
218
views
Row-stochasticity of the Jacobian matrix of a stationary distribution
Let $P_{\mathbf{p}}$ be a $n \times n$ row-stochastic matrix whose entries are a function of a probability vector $\mathbf{p} \in \mathbf{R}_{> 0}^n$, $\sum_i p_i = 1$ and define the following ...
- 113
1
vote
1
answer
363
views
Variation of Markov Chain Convergence Theorem
Assume the chain $\{X_n\}_{n\in\mathbb{N}}$ on the statespace $(S,\mathcal{F})$ (we may assume it is countable) is aperiodic, irreducible and positive recurrent. We denote with $\pi$ its (unique) ...
- 113
1
vote
0
answers
101
views
How to fit a stochastic matrix to given data.?
Given a data sequence of noisy observations of a 3-state Markov chain $X$ -- $y_1$,$y_2$,...$y_n$, with two transition matrices $A_1$ and $A_2$ corresponding to different regions (**) in the (unit) ...
2
votes
0
answers
970
views
Is the stationary distribution of this Markov chain uniform?
First, a little bit of background: Since 2012, Canada has decided to phase out the penny for its coinage system. Product prices may still use arbitrary cents, especially since prices do not typically ...
- 29
11
votes
2
answers
2k
views
Can ergodic theory help to prove ergodicity of general Markov chain?
I am a beginner in ergodic theory. I have read some lecture notes(such as this and this) about it in hope that I could find something which helps to prove the ergodicity of some Markov chain taking ...
- 221
3
votes
0
answers
158
views
Worst-Case Solution to (Stochastic) Matrix Inequality
EDIT: Some specific conjectures added.
This problem comes with an associated stochastic process, but I phrase everything as linear algebra in case somebody from a non-probability community has seen ...
- 31
2
votes
1
answer
903
views
Mixing time of a continuous time Markov chain with arbitrary rate matrix
I would like to calculate the mixing time of a continuous time starting from the rate matrix and not necessarily assuming that the time in between jumps have rate 1 - all I have is the (finite ...
- 229
0
votes
1
answer
207
views
On the inverse problem of Dobrushin
Dobrushin, in this paper, looked into the following problem. Suppose We are given a Markov kernel (conditional distribution) $P_{Y|X}$. Information theorist usually call $W$ a channel. It is known ...
- 1,109
1
vote
0
answers
46
views
Is there an effective algorithm for finding "minimal discovery times" for large graphs?
Consider a large, probably sparse graph with Markovian random walkers on it.
Define the discovery time as the expected time to first reach a vertex by
random walk from a uniform start. Are there ...
- 355
1
vote
2
answers
2k
views
Does a irreducible set of states necessarily need to be closed in a Markov chain?
I have come across two different definitions for a 'irreducible set of states' of a Markov chain.
Definition 1: A subset of states $A$ of a Markov chain is irreducible if it is possible to access (...
- 23
1
vote
0
answers
111
views
Conditional probabilities in epidemic model
I was contemplating an epidemic model where infection and recovery rates are determined by links. Here node $i$ is infected first and recovers at a rate $\mu_i$. For all other nodes, the recovery is ...
- 519
1
vote
1
answer
352
views
Double Markovity
Suppose we have a double Markov relation for three random variables $X$, $Y$ and $W$ as follows
$$X\to W\to Y,$$ and $$X\to Y\to W.$$
How to prove that there exist functions $f$ and $g$ such that
$$...
- 1,109
6
votes
0
answers
337
views
Maximal inequalities for square of partial sums
Let $S_n = \sum_{i \leq n} X_i$ be the partial sums of a nice sequence of random variables $X_i$. In my application, $X_i$ is a functional of a finite-state, irreducible, aperiodic Markov chain, so ...
- 1,069
1
vote
0
answers
92
views
Simultaneous multiple perturbations in Markov chain Monte Carlo
I'm coding a McMC algorithm for geophysical applications.
Using the Metropolis-Hastings scheme to accept/reject the proposed models is smth that i thought i completely understood, but i don't. To be ...
- 11
6
votes
1
answer
509
views
Approximating Markov chains by Brownian motion
I would like a result along the following lines to be true, but haven't been able to locate it in the literature; pointers would be welcome.
Let $X_{t}$ be a finite-state, irreducible, aperiodic ...
- 1,069
1
vote
0
answers
181
views
Nonstationary Markov chain maximal inequality
Let $X_i$ be a (finite-state, irreducible, aperiodic) Markov chain, not necessarily stationary. (That is, it doesn't start from the invariant distribution; I'm happy to have it be time-homogeneous if ...
- 1,069
1
vote
2
answers
1k
views
Gibbs sampler with linear constraints
My problem concerns the estimation of truncated multivariate normal distributions under constraints.
Let $X_1$ and $X_2$ two random variables following normal distributions $\mathcal{N_1}(m_1,\...
- 45
12
votes
3
answers
4k
views
How to explain "Feller process" to an undergraduate student?
I had to explain in informal terms what a Feller process was, to undergraduate students who understand Markov property, Poisson processes and such. It was easy to define Levy process as generalisation ...
- 519