# Tagged Questions

**0**

votes

**1**answer

37 views

### DTMC random walk model [closed]

For a discrete Markov chain random walk with p < 0.5 with state space S= {0,1,2..}
What is the stationary distribution?
I could use any help.
Thank you

**3**

votes

**1**answer

100 views

### Approximating Markov chains by Brownian motion

I would like a result along the following lines to be true, but haven't been able to locate it in the literature; pointers would be welcome.
Let $X_t$ be a finite-state, irreducible, aperiodic Markov ...

**1**

vote

**0**answers

83 views

### random walk with reflecting barriers [closed]

Consider a random walk on the line 1,...,d. You start at point 1. At each step you flip a coin: heads means go left, tails means go right. If you're at 1 and get a heads, just stay where you are (same ...

**4**

votes

**3**answers

457 views

### How to explain “Feller process” to an undergraduate student?

I had to explain in informal terms what a Feller process was, to undergraduate students who understand Markov property, Poisson processes and such. It was easy to define Levy process as generalisation ...

**2**

votes

**0**answers

30 views

### Markov decision processes: action set revealed at point of decision

I have a problem which looks like a finite horizon Markov decision process (MDP), except the action space at each time is revealed at the decision making point. There is no way to know before hand the ...

**5**

votes

**1**answer

327 views

### References for a physicist migrating to stochastic processes

I've studied "Markov Chains" - Norris and "Measure, Integral and Probability" - Capinski, Kopp. Now, I'm looking for a couple of books (or other references) that help me bridging these two topics. ...

**0**

votes

**0**answers

18 views

### Examples of POMDPs where the actions impact the transitions of the underlying markov Chain

I am not sure if the following is a legitimate question for this board.
I am looking for examples of Partially observed Markov decision processes (preferably infinite horizon, Discrete time, Discrete ...

**1**

vote

**2**answers

107 views

### Invariant measure of Euler-Maruyama Discretisation of an Ito diffusion

Let $(X_t)_{t \geq 0}$ be a diffusion process with dynamics governed by the stochastic differential equation
\begin{equation}
dX_t = b(X_t)dt + \sigma(X_t)dW_t, ~~ X_0 = x_0,
\end{equation}
where ...

**3**

votes

**0**answers

85 views

### Nonlinear Markov process

Consider the following nonlinear $\mathbb{R}$-valued stochastic recursive sequence:
$ X_{n+1} = F(X_n) + W_{n+1}, \quad (W_n)_{n\ge1} \stackrel{ \scriptsize \mathrm{i.i.d.} }{ \sim } \phi. $
How can ...

**0**

votes

**1**answer

216 views

### Generating independent random variable from two correlated random variables

Suppose two random variables $X$ and $V$ are given. I am wondering what kind of condition we need to impose on joint distribution of $V$ and $X$ to make sure that there exists a random variable $Z$ ...

**1**

vote

**0**answers

67 views

### CLT for a Markov Renewal Process

Suppose $(X,T)=\{(X_n,T_n)\}_{n\geq0}$ is a Markov renewal process, where $X$ is a finite-state, discrete-time Markov chain with state space $\{1,2,...,R\}$. $T$ is the additive component, more ...

**1**

vote

**1**answer

104 views

### Quasi-stationary distribution for a death process

In the paper, Survival in a quasi-death process by van Doorn and Pollett, the quasi-stationary distribution of a transient CTMC is discussed and QSD for a simple death process is derived.
Consider a ...

**2**

votes

**1**answer

132 views

### Transition probabilities in coupled Markov chains

I know that for a continuous-time Markov chain, the probability of transition from time $0$ to $t$ is given by $P(t)=e^{Q(t)t}$. I have a system of $N$ interdependent continuous-time Markov chains ...

**3**

votes

**0**answers

44 views

### Relative vulnerabilities in SIS epidemic model

Consider the SIS model of epidemic spreading. There is a finite graph $G(V,E)$, link infection rates $\lambda_{ij}$ and node recovery rates $\mu_i$. There are a few initial nodes which are infected at ...

**0**

votes

**1**answer

98 views

### Supremum in a Markov chain model

A Markov chain $X$ with finite state space $\{1,2,\cdots,N\}$ is defined on a probability space $(\Omega, P, \mathcal{F})$ equiped with filtration $\{\mathcal{F}_t\}$. And we assume that we can reach ...

**3**

votes

**4**answers

311 views

### Does the variance of a continuous time, time homogeneous, Markov process starting from one point necessarily not decrease?

Let $x_t$ be a zero mean, time homogeneous Markovian process (chiefly look at the case where the value is in $1$ dimension) over time $t$ starting from $x_0=0$. Is it necessary that, in continuous ...

**1**

vote

**0**answers

131 views

### The problem of the drunkard in a valley [closed]

We consider a Markov chain on a subset of positive integers S = {0, 1, 2, 3, .......N}, with transition probabilities defined as follows:
The chain jumps only one unit to the left or right.
p(i, j) ...

**3**

votes

**0**answers

178 views

### Generalized Markov Processes on CW complexes of dimension > 1

Markov processes have a large variety of applications to physics and chemistry (as well as many other fields). Such processes are formulated on graphs, i.e., CW complexes of dimension one. It is ...

**0**

votes

**1**answer

230 views

### Markov Chain: state reduction

Hi I am trying to understand a proof in a paper (written by Isaac Sonin), I don't know if anyone could give me a clarification on the following:
Firstly we have a Markov chain $\{Y_k\}$ with finite ...

**2**

votes

**0**answers

137 views

### Markov operators and existence of ergodic measures

My question refers to the yesterday's question (see here)
of John Learner and goes as follows:
Can we deduce the existence of an ergodic measure if we know that an invariant measure exists, but the ...

**0**

votes

**0**answers

103 views

### Inequality relating stationary probabilities and transition probabilities

Let $P$ be the transition probability matrix of a aperiodic irreducible DTMC and let $\pi$ be its stationary distribution. I would like to know if there is any literature on types of Markov chains ...

**1**

vote

**1**answer

87 views

### Empirical distribution of a collection of iid Markov chains

Suppose we have $N$ independent 2-point Markov chains each having a rate matrix $Q = [-1,1;1,-1]$ and stationary distribution $\pi = [0.5,0.5]$. At time $t=0$, we initiate the chains so that the ...

**1**

vote

**1**answer

490 views

### Hitting time probability in a Random Walk with possibility to die.

A Random Walker can move of one unit to the right with probability $p$, to the left with probability $q$ and it can jump again to the starting point with probability $r$ and die. Naturally $p+q+r=1$. ...

**1**

vote

**1**answer

205 views

### Extending Wald's equation to two classes of i.d. random variables?

I try to adopt Wald's equation to a slightly more complex problem. In fact, after a full day, I found some solution now, but it has a confusing argument in the middle. Perhaps somebody can help me at ...

**10**

votes

**2**answers

555 views

### Markov chains: invariant measures and explosion

The following seems like such an elementary question, but I didn't get anywhere with it.
Suppose you are considering a Markov chain in continuous time which is transient and has an invariant measure ...

**2**

votes

**1**answer

352 views

### MCMC with progressive demollification of delta distributions

Edit: I simplified the example to a canonical case for clarity.
Given an integral $\int_{\Omega}{g(\mathbf{x})}$ with a well-posed integrand $g(\mathbf{x})$ defined on some multidimensional space ...

**4**

votes

**0**answers

355 views

### The spectrum of a Markov Operator and Invariant Measures

Suppose I have a discrete-time Markov Chain (in an infinite dimensional state space $\Omega$) with Markov operator $P$, a linear operator on the space of bounded measurable functions on $\Omega$. (Or ...

**2**

votes

**1**answer

186 views

### Stochastic processes having Markov kernels

Let $(\Omega_1, \mathcal{F}_1, P_1)$ and $(\Omega_2, \mathcal{F}_2, P_2)$ be probability spaces and suppose $(X_t)$ and $(Y_t)$ are real-valued stochastic processes defined on the respective spaces. ...

**3**

votes

**1**answer

575 views

### Difference in probability distributions from two different kernels

Let $(E,\mathscr E)$ be a measurable space and $P,\tilde P$ be two stochastic kernels on that space. I wonder how the induced measures $\mathsf P_x$ and $\tilde{\mathsf P}_x$ differ on the space of ...

**1**

vote

**1**answer

175 views

### One point on $\phi$-irreducibility

Let $P(x,A)$ be a stochastic kernel on a measurable space $(E,\mathcal E)$ and $G = \sum\limits_0^\infty P^n$ be its potential kernel. A $\sigma$-finite measure $\phi$ is called the irreducibility ...

**3**

votes

**1**answer

364 views

### stochastic processes conditional on other stochastic processes

Problem: I'm working in reliability field and have seen papers written on the topic like process of failures when systems are functioning under unobservable (or observable) Markov-like environment, ...

**1**

vote

**1**answer

499 views

### Ergodicity of a Markov chain

Hi,
I'd appreciate some help on a Markov chain result I'm trying to show. I believe the following is sufficient for a continuous time Markov chain $(X_t)$ with a countable state space to be ergodic:
...

**3**

votes

**1**answer

313 views

### Probability-one event for Markov chain

Let $X$ be a Markov chain, with countable state space $I$ and transition probability matrix $P$. $X$ is irreducible, but need not be recurrent. Let $S$ be a fixed subset of $I$.
Define a subset $K$ ...

**0**

votes

**1**answer

138 views

### Modification of a Markov process on the real line

Consider a real-valued Markov process $X$ with a transition density $f(x,y)$, i.e.
$$
\mathsf P[X\in A|X_0 = x] = \int\limits_A f(x,y)\,dy.
$$
For this process I want to find
$$
u(x) = \mathsf P[X_n ...

**1**

vote

**2**answers

317 views

### How to determine a specific graph process is Markovian or not ?

Say, here is a min-degree graph process, which starts with G_0 = the complement of K_n. Given G_t, choose a vertex u of minimum degree in G_t u.a.r., then a vertex v not adjacent to u in G_t u.a.r. ...

**1**

vote

**1**answer

295 views

### A bjection between two stochastic processes

Let x(t) be a Markov process. We define the stochastic process y(t) such that :
y(t) = x(f(t))
f : T -> T
T is the parameter set of the process x(t).
If we ...

**11**

votes

**1**answer

446 views

### Random walk origin return monotinicity

Consider a Markov chain on $\mathbb{Z}^d$ with transition kernel $P$ for adjacent vertices (non-diagonal). Essentially this is a $d$ dimensional random walk with the probability of a transition ...

**2**

votes

**2**answers

445 views

### Spectral gap of a product of Markov processes

For $m \in [N] \equiv \{1,\dots, N\}$, let $Q^{(m)}$ be the generator of a (well-behaved) continuous-time Markov process on a finite state space $[n_m]$. Write $J \equiv (j_1,\dots,j_N) \in \prod_m ...

**2**

votes

**3**answers

469 views

### Markov random field with continuous index set

Hi
There's Markov random field (MRF) which, by my Wikipedia-based knowledge, is an extension of Markov chain. I'd like to think of it as going from 1D to higher dimensional spaces. Inherent in its ...

**1**

vote

**2**answers

534 views

### Borel-Cantelli Lemma on MCs (absorbing states)

hi, I'm sorry if the question is silly, but I couldn't get my head around it for a while now.
In Markov Chains (MC) proving that a state is either recurrent or transient is through Borel-Cantelli ...

**2**

votes

**0**answers

330 views

### How to bound the second largest eigenvalue of a transition matrix of a non-irreducible Markov chain?

I have found several bounds (e.g., Cheeger, Poincare) for the case that the Markov chain is irreducible and reversible, however my Markov chain has one absorbing state. Any bound would be helpful, but ...

**2**

votes

**2**answers

591 views

### Counterexample Markov process

Let $X$ be a homogeneous Markov process in a continuous time with value in the set $E$. Suppose that for some $T>0,x\in E, A\subset E$ we have
$$
P_x[X_t\in A] = 0
$$
for all $t\in [0,T]$ but
$$
...

**1**

vote

**1**answer

393 views

### Reachability for Markov process

Let $X$ be a Markov process (in continuous or discrete time) and define an event
$$
R(T,A) = (\exists t\leq T: X_t \in A).
$$
I have seen in one paper that
$$
\Pr[R(\infty,A)] = \sup\limits_{\tau} ...

**2**

votes

**1**answer

156 views

### scalar diffusions are reversible

It is well known that under mild assumptions a scalar diffusion $dX_t = a(X_t) dt + \sigma(X_t) dW_t$ with invariant probability distribution $\pi$ is reversible. This is indeed not true for ...

**4**

votes

**1**answer

631 views

### A simple problem in markov chains

I'm trying to understand a 1954 paper of Kubo intitled "Note on the stochastic theory of resonance absorption". The specific problem can be stated mathematically as follows: let $X(t)$ be a random ...

**2**

votes

**2**answers

938 views

### Is there MDPs (Markow Decision Process) which have a non deterministic optimal policy ?

I'm working on Markov Decision Process and I have not found yet an example of MDP that has a stochastic (non deterministic) optimal policy. Is there MDPs that have a stochastic optimal policy or is it ...