# Tagged Questions

**0**

votes

**0**answers

17 views

### Zeros of non-lipschitz functions (when noisy estimates are available only)

Given noisy (martingale difference) of a Lipschitz continuous function $f$ it is known how to compute zeros of it. It is the stochastic approximation approach (by Borkar, Kushner and Yin etc.). Is ...

**0**

votes

**0**answers

19 views

### Examples of POMDPs where the actions impact the transitions of the underlying markov Chain

I am not sure if the following is a legitimate question for this board.
I am looking for examples of Partially observed Markov decision processes (preferably infinite horizon, Discrete time, Discrete ...

**0**

votes

**1**answer

145 views

### Supremum in a Markov chain model

A Markov chain $X$ with finite state space $\{1,2,\cdots,N\}$ is defined on a probability space $(\Omega, P, \mathcal{F})$ equiped with filtration $\{\mathcal{F}_t\}$. And we assume that we can reach ...

**1**

vote

**1**answer

98 views

### Kalman filter with long term bias

I was reading about the Kalman filter and I do not understand how it should be used when our measurements have a long term offset like GPS location updates do.
As I understand, the Kalman filter ...

**0**

votes

**1**answer

253 views

### Anyone has Kushner's book “Introduction to stochastic control” 1971? I need a theorem from it

In a paper I'm reading, it refers to Theorem 8, Page 217 of the book
"Introduction to Stochastic Control" H. J. Kushner, New York: Holt, Reinhart, and Winston 1971. Unfortunately I don't have it and ...

**2**

votes

**1**answer

83 views

### Maximal probability of “infinitely often” over MDP

Let us consider a Markov Decision Process (MDP) with a Borel state space $X$. Often, the optimization problems over MDP involve optimization of some objectives dependent on the reward function
$$
...

**4**

votes

**0**answers

69 views

### Sufficiency of stationary policy for negative stochastic dynamic programming

Consider a Markov Decision Process with Borel state space $X$ and Borel action space $U$, like the one defined in the book "Stochastic Optimal Control: Discrete-time case" by Bertsekas and Shreve. All ...

**2**

votes

**1**answer

136 views

### Upper bound concerning Snell envelope

Consider, on a filtred probability space $ \left (\Omega, \mathcal F, \mathbb F , \mathbb P \right )$ where $ \mathbb F = \left(\mathcal F_ t \right )_ {t\geq 0}$ is filtration satisfying the usuual ...

**0**

votes

**0**answers

144 views

### Stochastic optimal control with no diffusion

Classical stochastic optimal control problem is to minimize functional
$$
J(u) = \mathbf E \int_0^T f(t,x_t,u_t)dt,
\tag{1}
$$
subject to SDE
$$
dx_t = b(t,x_t,u_t)dt + \sigma(t,x_t,u_t)dW_t, \quad ...

**3**

votes

**0**answers

147 views

### stochastic control / geometric mean

Consider the following problem:
Given $\Omega$ and $U$ two symmetric definite positive matrices, choose a matrix $K$ to minimize the expectation $x' \Omega x + x'K'UKx$ when $x$ follows the invariant ...

**1**

vote

**3**answers

433 views

### convergence in distribution of stochastic gradient descent.

The stochastic gradient descent algorithm where only a noisy gradient (zero mean noise) is used to update current estimate is known to converge almost surely to the minimizer. However, if one is ...

**4**

votes

**2**answers

1k views

### Commuting supremum and expectation

Given a one-parametric random function on a probability space $(\Omega,\mathcal F,\mathbb P)$:
$X:U\times\Omega\to \mathbb R \text{ and } (a,w)\mapsto X(a,w), \text{ with } \sigma(X(a))\subseteq ...