The tag has no usage guidance.

learn more… | top users | synonyms

-3
votes
0answers
21 views

How to make Markov Chain model from sequence of data using MATLAB? [on hold]

I have a sequence and from that I have to make Markov Chain Model in MATLAB. Markov Chain model considers only 1-step transition probabilities i.e. probability distribution of next state depends only ...
2
votes
2answers
122 views

The necessary sufficient condition for recurrence of a Markovian random walk

Suppose $\sigma_{1},\sigma_{2},...$are i.i.d random variables.$S_{0}=0$. Define $S_{n}=S_{0}+\sum_{i=1}^{n}\sigma_{i}$, then ${S_{n}}$ is a Markovian random walk. I want to figure out the necessary ...
-1
votes
1answer
32 views

Discrete time hidden markov process [closed]

I am dealing with a hidden Markov model for variable $X_{t+1}$ where $X_{t+1}$ = $\alpha_{t}$$X_{t}$ + $(1-\alpha_{t})$$Z_{t}$ $X_{t}$ is an indicator variable indicating wether an individual is ...
1
vote
1answer
116 views

Neat definition of Harris Ergodicity

I can't find any reference where the definition of Harris Ergodicity for Continuous time Markov processes is defined. a) What would be exactly the definition? b) What reference could be helpful? ...
5
votes
1answer
310 views

Does every (generalized?) Markov chain admit transition probabilities?

To pose the question let us start by recalling the following notions: Transition Probabilities. A transition probability matrix between two measurable spaces $(S,\mathcal{S})$ and ...
1
vote
1answer
37 views

Is there an easy way to convert a non-deterministic optimal policy to a deterministic optimal policy for a given MDP?

For a MDP (Markov Decision Process) is there an easy way to convert a non-deterministic optimal policy into a deterministic optimal policy? The trivial way will take ...
4
votes
0answers
120 views

Decay to stationarity in a random walk on the hypercube

Let $\mu$ be a probability distribution on $\mathbb F_2^n$. Consider the random walk $X_0,X_1,\ldots$ defined by $$\begin{aligned} X_0 &= 0\\X_{i+1}&=X_i + Z,\end{aligned}$$ where $Z\sim \mu$ ...
2
votes
0answers
151 views

Must rows of a transition matrix be distinct?

Is it true that for all continuous time Markov processes on a countable state space $S$, we have all rows of the transition matrix $\mathbf{P}_t$ are distinct for all time $t\in[0,\infty)$ ? This ...
3
votes
0answers
58 views

Most visited vertex in a random walk with place dependent drift

Consider the following Markov chain on $\mathbb{Z}$: $$ P(x,x+1)=1-P(x,x-1)=\frac{1}{2}+e^{-|x|}\cdot \mathbf{1}_{\{x\neq 0\}} $$ Do there exist constants $c,C>0$ such that $$ c\cdot P^t(z,z) ...
1
vote
1answer
56 views

Choose uniformly from fixed-length paths in $[0,n]\cap\mathbb{Z}$ with fixed start and end

Let $X_k$ be a symmetric (discrete time) random walk on $\mathbb{Z}$ and let $m,n\in\mathbb{N}$. I want to chose uniformly from the paths of $X_k$, which start at $0$ stay in $[0,n]\cap\mathbb{Z}$ ...
0
votes
0answers
28 views

Sufficient moment conditions to make $E[\sup_n |X_n|]< \infty$ for Markov process $X_n$

Is there any Markov process $X_n$ for which we can impose sufficient moment condition which will imply $E[\sup_n |X_n|]< \infty$
0
votes
0answers
32 views

Strong Markov vector-valued process from component strong Markov process and independence

I want to prove that if $X$ and $Y$ are (continuous time) independent strong markov $\mathbb{R}$-valued processes w.r.t. their natural filtrations $\mathcal{F}^X_t$ and $\mathcal{F}^Y_t$, that the ...
1
vote
0answers
46 views

Expected number of forward jumps to reach a given quantile of a rv [closed]

I'm a noob in randomized algorithm and ran into a problem(definitely not home work. I'm doing a self study out of my interest with help of my friends. I'm pursuing research career in a machine ...
0
votes
1answer
29 views

Does an infinite chain of a.s. eventual transitions between states necessarily implies a.s transitions along the whole chain?

Given a Markov process among a (possibly infinite) set of states $S$, with possibly infinite depth (that is, the transition probabilities from $s_i \to s_j$ at time $t$ are permitted to depend not ...
2
votes
1answer
98 views

Random Walk 2D with dependent weights [closed]

I have spent a lot of time trying to solve this problem but have had no luck so far! Any help would be highly appreciated! Suppose I have a 3x3 grid as shown below. (3,1) (3,2) (3,3) (2,1) (2,2) ...
0
votes
0answers
50 views

Maximal inequality for Markov process

For a Markov process $\{X_n\}$ is there any inequality available for $$ E[\sup_{0 \leq n \leq k} X_{n}]$$ in terms of moments of $E[X_n], 0 \leq n \leq k$
0
votes
0answers
16 views

Can MDPs over functions be solved?

I understand that dynamic programs are difficult to be solved in general. However I have an MDP, for which intuitively I have a solution, I am curious to know if there is a formal approach to get a ...
-2
votes
1answer
32 views

how to resolve the infinite nesting of interactive POMDP

I am reading papers about I-POMDP. I cant understand the finitely nested I-POMDPs given in these papers. The belief update of the algorithm has a problem that agents' belief updates mutually depend ...
1
vote
0answers
56 views

Markov Chains and Simple Machine Learning [closed]

Suppose I have a large training set consisting of many strings of symbols. $TS = \{Str_0, Str_1, ..., Str_n\}$ $Str_i = \{Sym_0 ... Sym_{len}\}$ These strings of symbols are each generated by the ...
1
vote
0answers
23 views

Properties of a map regarding the space of invariant probability measures for controlled Markov process

Let us consider a controlled Markov process with the transition kernel $p(dy|x,\theta)$ ($\theta$ being the control parameter. Now, consider the map $\theta \to I(\theta)$ where $I(\theta)$ is the ...
0
votes
0answers
30 views

Markov Modulated Markov Chain

Consider a discrete time Markov chain $X_t$ on some finite state space $\mathcal{S}$ with transition matrix $P$. Now consider a process $Y_t$ also on $\mathcal{S}$, which conditioned on $X_{t}=s$ ...
2
votes
1answer
66 views

Eigenvectors of a perturbed reducible stochastic matrix

Let $Q$ be a $n\times n$ reducible stochastic matrix. Let $J$ be such that $[J]_{ij}={1 \over n}$. Now for a small positive constant $\alpha\in [0,1]$, consider the matrix ...
0
votes
0answers
56 views

Circular process ergodic?

Let us define a continuous-time Markov process on a circle consisting of $m-$ equally spaced points, i.e. every point has two neighbours. Now, we define a space of functions $S:= ...
0
votes
1answer
134 views

Mixing time of lazy random walk on the directed cycle $C_n$

Briefly: A hint (if this is easy), reference or derivation would be of great help. The question Let $C_n$ be the directed cycle with loops in each of its $n$ vertices, and consider the random walk ...
1
vote
0answers
15 views

Bounding Hidden Markov model Bayesian filter error with inexact models

In context of a hidden Markov model, I am interested in bounding the error of a Bayesian filter when using inexact state transition and observation models. Consider a hidden Markov model (HMM) with ...
0
votes
0answers
48 views

Regularity of the entrance measure of SRW

Let $S(n)$ be the discrete sphere of radius $n$ (i.e., the internal boundary of the Euclidean discrete ball $B(n)$) centered in the origin, and consider a simple random walk starting at some ...
0
votes
0answers
61 views

Order statistic of Markov chain sample path and related probabilities

Consider a 1D sample path, denoted as $\{X(1), ..., X(t), ..., X(n)\}$, generated from a discrete time finite state (time homogeneous) Markov chain over states $\{1,...,m\}$, with transition ...
1
vote
0answers
26 views

steady state of a continuous-time birth-death process

we consider a continuous-time birth-death process $\{X(t),t\geq 0\}$ with discrete state space taking non0negative integer values $\{0,1,2,3,...\}$. The transition rates of the process $\{X(t)\}$ are ...
3
votes
1answer
83 views

Similarity transformation of transition matrix of reversible Markov chain (reference request)

If $P$ is the transition matrix of a reversible Markov chain, and $\pi$ is its stationary distribution, and let $R$ be defined by: $$R_{ij} = \sqrt{\frac{\pi_i}{\pi_j}}P_{ij}~.$$ By reversibility, ...
2
votes
0answers
40 views

Spectral gap of two step Markov chain

Suppose $X_1,X_2,\ldots$ is a reversible Markov chain with state space of size $k$ and absolute spectral gap $\gamma_*.$ What is the spectral gap of the (non-reversible) Markov chain $Y_1,Y_2,\ldots,$ ...
0
votes
0answers
28 views

steady state distribution for a jump Markov chain

Consider a queueing process with the following transition matrix: $\mathbf{P}=\left( \begin{smallmatrix} 1-\lambda & \lambda & & & & & & &\\ \mu & ...
0
votes
1answer
86 views

Finite hitting time implies hits at any finite time?

I was wondering about the following problem: Assume we have a state space $S:=\mathbb{Z}$ and a Markov chain, such that we can go from any state $x$ to some state $y$ with positive probabilities, ...
0
votes
0answers
26 views

steady state distribution of the following infinite-state Markov chain

Given the following state transition equation: $P_0(n+1)=P_0(n)(1-\lambda \Delta t)+ P_1(n)\mu \Delta t$ $P_j(n+1)=P_{j}(n)(1-\lambda \Delta t-\mu \Delta t)+\lambda \Delta t P_{j-1}(n)+ \mu \Delta ...
0
votes
1answer
110 views

Markov chain with Feller property

Does anybody know whether there is an analysis of when the monotone decreasing chain has the Feller-property? The monotone decreasing is defined as a chain on $\mathbb{N}$ and the rate of going down ...
0
votes
0answers
130 views

Calculate the KL divergence between two transition matrices

I want to calculate how different two markov transition matrices are. For example: $\begin{pmatrix} .2 & .8 \\ .1 & .9 \end{pmatrix}$ and $\begin{pmatrix} .3 & .7 \\ .1 & .9 ...
3
votes
0answers
39 views

Memorylessness of residence times for a Markov process

I'm stuck on the trivial problem of showing memorylessness of holding (residence) times for a continuous time homogeneous Markov chain on finite state space. I have a homogeneous Markov process ...
8
votes
1answer
153 views

Equalizing Geometric means of Graph Cycles

Consider a strongly connected directed graph $G$. I have been stuck on the following question: can you assign real numbers in $[0,1]$ to each edge of $G$ so that the geometric mean of all cycles are ...
0
votes
0answers
42 views

positive Harris recurrent, aperiodic, stationary Markov chain

How to proof that every positive Harris recurrent, aperiodic, stationary Markov chain is alpha-mixing (strong-mixing)?
1
vote
1answer
130 views

Two types of random walkers on square lattice

Consider a two dimensional square lattice ($n$ by $n$), which is our space $S$ (each point labelled by an index $1\to n^2$), containing two types of particles, distinguished here by either an index ...
3
votes
1answer
103 views

Markov-semigroup sobolev inequality

I have a question about the following definition: A probability measure $\mu$, such that the Markov semi-group $e^{Lt} \in L(L^2)$ exists and is symmetric, satisfies the Sobolev inequality iff for ...
3
votes
1answer
88 views

How much larger than the relaxation time can the mixing time be?

The notation is mostly taken from the book "Markov chains and mixing times" by Levin, Peres, and Wilmer. Consider an irreducible, aperiodic, time-reversible, discrete-time Markov chain on a finite ...
1
vote
1answer
69 views

General Markov Chains on same Probability Space?

A Markov chain $(X_i)_{i\in \mathbb{N}}$ on a measurable space $(E,\Sigma)$ is (see e.g. Revuz or Meyn/Tweedie) constructed on the following probabilty space. $$ \Omega = \{ (x_l)_{l \in \mathbb{N}} ...
0
votes
0answers
37 views

Validating a probability density distribution forecast model for a Markov process

Let's say we have a Markov process $X_t$, and we come up with a forecast model that takes some information from outside world and says: "value $X_{t+1}$ has probability density distribution $P_t(x)$". ...
0
votes
0answers
64 views

Markov chain matching local time

Let $\left(X_{t}\right)_{t\geq0}$ be a Markov process taking values in a finite state space $E$. Its local time at $y\in E$ started at $x\in E$ is defined as $$ ...
3
votes
0answers
108 views

The spring Markov chain on $\mathbb{N}$

I'm trying to understand and learn more about "almost surely bounded" Markov chains on countable state spaces. I'm looking for references where I can learn how to work with more complicated examples ...
0
votes
0answers
39 views

The effect of a single Markov transition on fidelity

Let $p$ and $q$ be two probability vectors of length $n$. The fidelity (or Bhattacharyya coefficient) of $p$ and $q$, is $$ F(p,q) \ := \ \sum_{i=1}^n \sqrt{p_i \cdot q_i}. $$ Let $A$ be a ...
0
votes
0answers
44 views

Ergodicity property for continuous-time Harris positive Markov process

I have posted this question on there, but got no answer. The following theorem is Theorem 13.3.3 of Meyn and Tweedie's Markov Chains and Stochastic Stability on page 328: Theorem 13.3.3. If ...
2
votes
0answers
67 views

Customers and Anti-Customer Queueing Problem: What is the Customer delete probability

Hello may I ask for your help? First the setting: I have got a problem with some queueing theory. The whole problem would be a grid of nodes, all nodes have an operation intensity $\mu_{i,j}$. ...
2
votes
1answer
184 views

Stationary distribution for time-inhomogeneous Markov process

I have a two state, discrete time, time-inhomogeneous Markov process with transition matrix defined by $$T_i=\begin{pmatrix} 1-p_i\alpha & p_i\alpha \\ p_i\beta& 1-p_i\beta \end{pmatrix}$$ ...
1
vote
1answer
126 views

Row-stochasticity of the Jacobian matrix of a stationary distribution

Let $P_{\mathbf{p}}$ be a $n \times n$ row-stochastic matrix whose entries are a function of a probability vector $\mathbf{p} \in \mathbf{R}_{> 0}^n$, $\sum_i p_i = 1$ and define the following ...